Step-by-Step Teaching

Step 1: The Core Problem — Why Counting Votes Is Not Enough

The Intuitive Setup

Imagine you ask 10 LLM judges whether a retrieved passage is relevant to a user's question:

Judge 1:  ✅ Relevant
Judge 2:  ✅ Relevant
Judge 3:  ✅ Relevant
...
Judge 8:  ✅ Relevant
Judge 9:  ❌ Not Relevant
Judge 10: ❌ Not Relevant

Result: 8/10 say Relevant → Majority Vote = Relevant

8 out of 10 feels convincing. But should it be?

The Hidden Danger

Ask yourself: Why did those 8 agree?

🔑 Key Insight: The number of agreeing votes matters less than the independence of those votes. Correlated agreement inflates your confidence without adding real information.

Analogy to Make It Concrete

Think of witness testimony in a courtroom:

• 8 witnesses who each independently saw the same event = very strong evidence
• 8 witnesses who all watched the same video recording together and then testified = essentially one piece of evidence repeated 8 times

The same logic applies to LLM judges.

---

Step 2: Why Standard Approaches Fall Short

Majority Vote (Baseline 1)

Every judge gets 1 vote → most votes wins

• ✅ Simple and interpretable
• ❌ Treats all judges as equally reliable
• ❌ Assumes errors are independent across judges

Weighted Majority Vote (Baseline 2)

More accurate judges historically → get more influence

• ✅ Better than uniform voting
• ❌ Still assumes errors are independent across judges

The Shared Fatal Flaw

Both methods assume that when judges are wrong, they are wrong for different reasons. In reality, LLM judges often fail together because they:

• Share the same prompt template
• Belong to the same model family
• Were fine-tuned on similar data
• Have the same blind spot for certain phrasing

🔑 Key Insight: When errors are correlated, a majority can be less informative than it appears. Standard methods have no mechanism to detect or correct for this.

---

Step 3: The Solution — Treating the Panel as a Network

Shifting the Mental Model

Instead of thinking of judges as independent voters, think of them as nodes in a network where some pairs are more connected than others.

Old Model:          New Model:
                    
J1  J2  J3         J1 ——— J2
|   |   |           |  \  /  |
✅  ✅  ✅          |   \/   |
                    |   /\   |
(independent)      J3 ——— J4
                   (with relationships)

What the Network Captures

Each judge still has its own reliability profile (how often it's right), but now pairs of judges also have relationships:

• Some pairs agree more often than their individual accuracy would predict → they may share a bias
• Some pairs provide complementary perspectives → their agreement is more meaningful

---

Step 4: The Ising Model — How Dependence Is Modeled

What Is an Ising Model?

An Ising model is a statistical framework originally from physics, used to represent pairwise dependencies between binary variables.

In this context:

• Each judge produces a binary output (relevant / not relevant, toxic / not toxic, etc.)
• The model learns two things simultaneously:
  1. Each judge's individual skill/reliability
  2. The pairwise similarity between every pair of judges

Parameters learned:
├── Judge skill:       How accurate is Judge A on its own?
├── Judge skill:       How accurate is Judge B on its own?
└── Pairwise coupling: Do A and B agree more than their 
                       individual skills would predict?

Two Variants of the Model

Example of class-dependent behavior: Judges broadly agree on clear-cut relevant passages, but split into recognizable clusters on ambiguous ones. The class-dependent model captures this nuance.

---

Step 5: How the Algorithm Learns (Without Human Labels)

This is an unsupervised method — it learns purely from judge outputs, without needing human-annotated ground truth for training.

The Learning Loop

Step 1: Start with initial parameter guesses
           ↓
Step 2: Combine current votes to estimate 
        probability that each item's true label is positive
        (these are "soft guesses," not hard labels)
           ↓
Step 3: Use those soft probabilities to re-estimate:
        - Each judge's reliability
        - Pairwise dependence between judges
           ↓
Step 4: Repeat Steps 2–3 until estimates stabilize
           ↓
Step 5: Use human reference labels ONLY to measure 
        final accuracy (not for training)

🔑 Key Insight: The algorithm treats the true label as a hidden variable it infers jointly with judge reliability and dependence. It extracts signal from the pattern of agreements and disagreements, not just the vote counts.

Why This Matters Practically

Teams that already run LLM-as-a-judge pipelines at scale are already collecting this data. Evaluation logs contain rich patterns of agreement and disagreement that standard majority voting completely ignores. Dependence-aware aggregation turns those existing logs into a stronger signal — no additional labeling required.

---

Step 6: Experimental Results — Does It Actually Work?

Setup

• 3 tasks: Relevance classification, toxicity classification, summarization assessment
• 10 judge models, all run at temperature zero (deterministic outputs)
• Compared against: weighted majority vote and uniform majority vote

Results

What the Numbers Tell Us

• Improvements range from 9% to 14% over the best baseline
• The gains are largest where judge dependence is highest (e.g., summarization, where judges likely share rubric interpretations)
• The method needs enough data and enough judges to estimate meaningful relationships — it is not a silver bullet for tiny panels

---

Step 7: Best Practices — What to Do Differently

1. Evaluate the Panel, Not Just Individual Judges

A set of individually strong judges can still be collectively redundant if they fail in the same way.

❌ Don't ask: "Is each judge accurate?"
✅ Do ask:    "Does each judge add independent information?"

2. Treat Model Diversity as Statistical Diversity

Mixing model families or architectures only helps to the extent that it changes error patterns. Two models from the same family with the same fine-tuning may be diverse in name only.

3. Inspect Agreement Structure

Strong clusters in the agreement network reveal:

• Shared rubrics or prompt templates
• Shared model behavior
• Task-specific ambiguity

This information is valuable even when the final label doesn't change — it tells you how much to trust your confidence estimate.

4. Report Uncertainty With Dependence in Mind

❌ 10 correlated votes   → should NOT produce same confidence as
✅ 10 independent votes  → genuine high confidence

Confidence intervals and uncertainty estimates should reflect the effective number of independent opinions, not the raw vote count.

---

Summary: The Complete Mental Model

QUESTION: When LLM judges agree, should we believe them?

ANSWER: It depends on WHY they agree.

├── Agreement from independent sources
│   └── Strong signal ✅
│
└── Agreement from correlated sources (shared prompts, 
    model families, training data, blind spots)
    └── Inflated confidence ❌

SOLUTION: Dependence-aware aggregation
├── Model the judge panel as a network
├── Learn pairwise dependencies from evaluation logs
├── Adjust aggregate scores to discount redundant agreement
└── Result: 9–14% accuracy improvement over weighted majority vote

🔑 The core lesson: When LLM judges agree, always ask why. Sometimes agreement is independent evidence. Sometimes it is a shared blind spot. A good aggregation method — and a good practitioner — should be able to tell the difference.