MQM Framework: The Complete Guide to Translation Quality Evaluation

Multidimensional Quality Metrics (MQM) is a framework for evaluating translation quality developed through the European Union's QTLaunchPad and QT21 research projects, led by the German Research Center for Artificial Intelligence (DFKI). Unlike single-score evaluation methods, MQM provides a structured taxonomy of error types that enables granular, reproducible quality assessment.

The framework emerged from a recognition that existing quality evaluation methods — including LISA QA, SAE J2450, and ad hoc rubrics — were either too narrow in scope or too inconsistent in application. MQM unified these approaches into a single, extensible hierarchy that can be customized for specific use cases while maintaining comparability across projects.

At its core, MQM operates on a simple principle: quality is measured by the absence of errors. Rather than asking evaluators to assign subjective quality scores, MQM asks them to identify, classify, and weight specific issues in a translation. This approach reduces inter-annotator disagreement and makes quality scores actionable — you know exactly what needs to be fixed.

The framework has been adopted by major language service providers, technology companies, and standards bodies. It forms the quality metric backbone for the TAUS Dynamic Quality Framework and has influenced the development of ISO 5060 (Translation Quality Evaluation).

MQM uses three severity levels to weight errors. Critical errors (typically 25 penalty points per instance) represent issues that could cause harm, legal liability, or severe misunderstanding. Examples include mistranslated safety warnings, wrong drug dosages, or offensive content introduced by translation. A single critical error usually results in a failing score regardless of overall quality.

Major errors (typically 5 penalty points) impair the meaning or usability of the translation but don't cause harm. An omitted sentence in a user manual, a consistently wrong technical term, or grammar errors that change meaning fall into this category. Multiple major errors indicate a translation that needs significant revision.

Minor errors (typically 1 penalty point) are noticeable but don't impair understanding. Inconsistent capitalization, minor style deviations, or awkward but comprehensible phrasing are minor errors. While individually insignificant, a high density of minor errors indicates poor overall craftsmanship.

The standard scoring formula is: Score = 100 - (Sum of Weighted Penalties / Word Count * Normalization Factor). This produces a score from 0 to 100, where 95+ is generally considered passing quality for professional publication. The normalization factor adjusts for text length — a 50-word text with one major error should not receive the same score as a 5,000-word text with one major error.

Organizations can customize penalty weights to match their quality requirements. A pharmaceutical company might assign 50 points to terminology errors (double the default) while a creative marketing agency might weight style errors more heavily. The framework's flexibility is one of its greatest strengths.

Inter-annotator agreement remains MQM's biggest challenge. Studies show that even trained annotators disagree on error classification 20-40% of the time, particularly for fluency and style dimensions where judgment is inherently subjective. Two qualified reviewers can assign different severity levels to the same error, leading to score variation.

MQM requires trained evaluators. The framework's granularity is both a strength and a barrier to adoption — classifying errors into the correct subcategory requires familiarity with the taxonomy and domain expertise. Untrained evaluators tend to over-flag minor issues or miss subtle accuracy errors.

Domain-specific weighting is essential but not standardized. The default penalty weights may not reflect actual quality priorities for all content types. Medical translation should weight accuracy errors far more heavily than marketing translation, but MQM doesn't prescribe domain-specific configurations out of the box.

MQM measures translation errors, not translation excellence. A score of 100 means zero errors were found — it doesn't mean the translation is elegant, creative, or optimally adapted. For transcreation, literary translation, or marketing copy where creative quality matters, MQM should be supplemented with other evaluation methods.

Despite these limitations, MQM remains the most rigorous and widely adopted framework for translation quality evaluation. Its structured approach makes quality measurable, comparable, and improvable — which is why it has become the industry standard for professional translation assessment.