After studying this material, students should be able to:

1. Explain what LLMs are and their role in Machine Translation (MT) evaluation
2. Describe the two main MT quality evaluation schemes: MQM and ESA
3. Compare LLM performance vs. human annotator performance in MT evaluation
4. Identify specific challenges faced by both humans and LLMs in annotation tasks
5. Evaluate the reliability of LLMs as alternatives to human judgment across different contexts
6. Understand the concept of inter-annotator agreement and why it matters

---

Step-by-Step Study Material

Step 1: Foundation — What is Machine Translation Evaluation?

What is Machine Translation (MT)?

Machine Translation is the automatic conversion of text from one language to another using software (e.g., Google Translate, DeepL).

Why Do We Need to Evaluate MT?

MT systems are not perfect. We need to measure:

• How accurate the translation is
• How fluent it reads
• What errors it contains

🔑 Key Insight: Evaluation is critical because poor translations can cause miscommunication, especially in high-stakes domains like medicine or law.

---

Step 2: Who Evaluates MT Quality?

Traditionally, human annotators (trained linguists or bilingual speakers) review translations and score them.

The Problem with Human Evaluation:

The Proposed Solution: Use LLMs

Large Language Models (LLMs) like GPT-4 are being explored as a faster, cheaper alternative to human annotators.

🔑 Key Question the paper asks: Can LLMs reliably replace humans for MT evaluation?

---

Step 3: Understanding the Two Evaluation Schemes

Scheme 1: Multidimensional Quality Metrics (MQM)

What it is: A detailed, structured framework that categorizes translation errors into specific types.

How it works:

• Annotators identify errors and assign them to categories
• Examples of error categories:
  • Accuracy (wrong meaning)
  • Fluency (awkward phrasing)
  • Terminology (wrong technical term)
• Each error is also rated by severity: Minor, Major, or Critical

Example:

Source (English):  "The patient requires immediate surgery."
Translation (FR):  "Le patient nécessite une chirurgie urgente." ✅
Bad Translation:   "Le patient veut une chirurgie." ❌
                    → Accuracy Error (Major): "requires" ≠ "wants"

🔑 MQM is fine-grained — it demands detailed, precise judgment. This makes it harder for both humans and LLMs.

---

Scheme 2: Error Span Annotation (ESA)

What it is: Annotators highlight specific spans (portions of text) that contain errors, without necessarily categorizing them in detail.

How it works:

• Read the translation
• Mark the exact words/phrases that are wrong
• Assign a severity level

Example:

Translation: "The cat sat on the [mat yesterday]"
                                  ↑ Error Span highlighted

🔑 ESA focuses on locating errors rather than deeply classifying them. It sounds simpler but still requires precision.

---

Step 4: What is Inter-Annotator Agreement?

Definition

Inter-annotator agreement (IAA) measures how consistently different annotators give the same judgment on the same translation.

Why It Matters

• High agreement = the evaluation is reliable and consistent
• Low agreement = the evaluation is subjective or unclear

How It's Measured

Common metrics include:

• Cohen's Kappa (κ) — measures agreement beyond chance
• Correlation scores — measure how similarly annotators rank translations

🔑 The paper compares:

• Human vs. Human agreement
• LLM vs. Human agreement

---

Step 5: What the Research Found

Finding 1: LLMs Sometimes Match or Exceed Human Agreement

LLM-Human Agreement > Human-Human Agreement
         ✅ (for SOME tasks)

This is promising — but it's not the full picture.

---

Finding 2: Performance Varies Dramatically

Both humans and LLMs perform inconsistently across:

🔑 Conclusion: You cannot assume an LLM that works well for English-French news translation will work equally well for Swahili-Japanese medical translation.

---

Finding 3: Humans and LLMs Struggle with DIFFERENT Things

Breaking this down:

Human Challenges:

• Fine-grained MQM: Deciding if an error is "Accuracy/Mistranslation" vs. "Accuracy/Omission" requires deep expertise
• Low-resource languages: Fewer training resources = less confident annotators

LLM Challenges:

• Minor errors: Small mistakes (punctuation, slight word choice) are often missed
• Wrong language variants: Confusing British vs. American English, or Brazilian vs. European Portuguese
• Error span annotation: Precisely locating where in the text the error occurs

---

Step 6: The Bigger Picture — Implications

Why This Matters for AI Development

Training Data Quality → Model Performance
        ↑
   Annotation Quality

If annotation is unreliable → training data is noisy → AI models perform worse.

The Proposed Solution: Human-LLM Collaboration

Rather than choosing either humans or LLMs, the paper suggests a collaborative pipeline:

Step 1: LLM performs initial annotation (fast, cheap)
         ↓
Step 2: Human reviews LLM output (focused effort)
         ↓
Step 3: Combined judgment = higher reliability

🔑 This leverages the strengths of both while compensating for their individual weaknesses.

---

Step 7: Summary Table

---

Quick Knowledge Check ✅

Test yourself:

1. What does MQM stand for and what makes it difficult?
2. How does ESA differ from MQM?
3. Name two things LLMs struggle with in MT annotation.
4. Why is inter-annotator agreement important?
5. What does the paper recommend as the best path forward?

---

💡 Core Lesson: Neither humans nor LLMs are perfect MT evaluators. The path to reliable evaluation lies in understanding where each fails and designing systems that combine their complementary strengths.