After studying this material, you should be able to:

1. Explain why simple production metrics (acceptance rate, thumbs-up) fail as ground truth for evaluating AI code reviewers
2. Describe how DashBench works as a multi-signal measurement framework
3. Interpret precision, recall, F1, and weighted variants in the context of code review evaluation
4. Analyze tradeoffs between staged vs. single-pass architectures using benchmark data
5. Apply the principle that no single metric is sufficient when evaluating complex AI systems

---

Step-by-Step Study Material

Step 1: Why Measuring AI Code Review Is Hard

The Tempting (Wrong) Approach

The most obvious way to measure an AI code reviewer is to watch what engineers do with its comments in production:

• Did the engineer accept the comment? → Looks like a win
• Did the engineer reject the comment? → Looks like a failure

This feels intuitive. But it has a fundamental structural flaw.

The Confusion Matrix Problem

A confusion matrix has four cells:

                    AI Said: Issue    AI Said: Nothing
Actually an issue:  True Positive     FALSE NEGATIVE ← INVISIBLE
Actually fine:      False Positive    True Negative  ← INVISIBLE

Acceptance-based measurement only ever fills two cells:

Key Insight: You can never know what the AI missed by watching what engineers accept. Silence is unobservable.

Human Acceptance Is Not Ground Truth

Engineers accept or reject comments for reasons unrelated to correctness:

• PR urgency or timing
• How invasive the fix would be
• Ownership context
• The issue was already handled elsewhere

This makes acceptance useful product telemetry but a weak benchmark label.

---

Step 2: Understanding the Core Metrics

Before reading DashBench results, you need to understand the metrics it uses.

Precision vs. Recall

Think of a code reviewer casting a net to catch bugs:

PRECISION = Of all the issues the AI flagged, how many were real?
           = True Positives / (True Positives + False Positives)
           = "When it speaks, is it right?"

RECALL    = Of all the real issues that existed, how many did the AI find?
           = True Positives / (True Positives + False Negatives)
           = "How much did it catch?"

The tension between them:

Why Weighted Metrics?

Not all bugs are equal. Missing a critical security flaw is far worse than missing a low-severity style issue.

DashBench applies severity weights:

Weighted recall therefore penalizes missing a critical bug 8× more than missing a low-severity one.

Why this matters: A system could look good on raw recall by catching many trivial issues while missing every critical bug. Weighted metrics expose this.

---

Step 3: How DashBench Works

DashBench is a measurement layer that replays historical PRs and scores whether the AI found real, human-actionable findings.

The Core Loop

Historical PRs
      ↓
  Replay through AI reviewer configurations
      ↓
  AI produces structured findings (severity + evidence + file anchors)
      ↓
  Compare against adjudicated ground truth labels
      ↓
  Score: precision, recall, F1, cost, latency
      ↓
  Disagreement audit → feeds back into judge calibration

The Three Case Types

DashBench deliberately includes different PR types to stress-test different behaviors:

How Labels Are Built (The Hard Part)

Most benchmarks take a shortcut: they use human labels as ground truth. DashBench refused to do this.

Three sources were compared against each other:

1. Engineer annotations (the PR authors)
2. Original candidate findings
3. An agentic judge (LLM-based)

Where they disagreed, the team manually re-reviewed the evidence and resolved it by hand. Those resolved cases then became calibration data for the judge.

Key Principle: Ground truth doesn't rest on any single fallible source. Human judgment, production feedback, and agentic evaluation each contribute — none is treated as infallible.

---

Step 4: The Staged Architecture Being Measured

DashBench measures a staged (scout + reviewer) architecture against simpler single-pass baselines.

How Staging Works

SINGLE-PASS:
  Diff → [One LLM] → Comments

STAGED:
  Diff → [Scout: flags suspicious areas]
              ↓
         [Deep Reviewer: investigates leads, confirms or drops claims]
              ↓
         Final Comments

This mirrors how strong human reviewers work:

1. Form hunches about risky areas
2. Aim attention at those areas
3. Validate before raising concerns

Why This Architecture Is Good for Measurement

Because every component is a variable you can hold fixed or swap independently:

• Scout model
• Reviewer model
• Context policy
• Tool policy
• Runtime budget

This means DashBench can answer: "For this specific combination, what tradeoff between coverage, precision, cost, and latency do we get?"

---

Step 5: Reading the Benchmark Results

Does Staging Buy Coverage?

What this tells us:

• Staging dramatically increases recall (53.6% vs. 30.7% vs. 20.2%)
• Precision stays roughly similar across all three
• The cost is real: staging costs ~5× more per PR
• This is the honest tradeoff DashBench is designed to surface

Which Model Mix Wins?

No single configuration dominates every axis:

Key Insight: "Best model" is a meaningless question until you specify best at what. DashBench makes the tradeoffs explicit rather than hiding them behind a single score.

Does Severity Change the Story?

Yes — significantly. Here is recall broken down by severity:

Observations:

• GPT 5.5 high (no scout) is surprisingly competitive on high-severity issues despite being cheap
• It almost completely misses low-severity issues (4.2% recall)
• Configurations that look similar in aggregate miss very different kinds of issues

This is exactly why one aggregate score is insufficient.

---

Step 6: The Four Core Lessons

Lesson 1: Trust Must Be Earned Through Triangulation

No single signal is reliable ground truth:

Human labels     → Reviewers miss valid issues, accept weak claims
Production data  → Can't observe false negatives
Agentic judge    → Non-deterministic, needs calibration

Solution: Use all three, treat none as infallible

Lesson 2: Humans Don't Scale for Labeling

Human attention degrades on complex tasks. AI judging is what makes labeling at scale tractable — but only with:

• Calibration data from resolved disagreements
• Audit sets
• Stable rubrics

Lesson 3: Variance Is a Feature, Not a Bug

LLMs are non-deterministic. This has two implications:

Design your measurement system for stochasticity rather than pretending it away.

Lesson 4: When One Metric Can't Work, Use Many

These metrics move independently:

Weighted Precision  ←→  Weighted Recall
Weighted F1         ←→  Cost per PR
High/Critical Recall ←→  Low Severity Recall
Latency             ←→  Coverage

A system can win on one and lose on another in the same run. "Better" requires specifying better at what.

---

Step 7: Putting It All Together — The Mental Model

Here is how all the concepts connect:

PROBLEM: How do we know if our AI code reviewer is actually good?

WRONG ANSWER: Watch acceptance rates
  → Only sees 2 of 4 confusion matrix cells
  → Can't observe false negatives (missed bugs)
  → Human acceptance ≠ correctness

RIGHT ANSWER: Build a benchmark (DashBench)
  → Replay frozen historical PRs
  → Compare against multi-source adjudicated labels
  → Score with multiple metrics (precision, recall, F1, weighted variants)
  → Measure cost and latency too
  → Never trust any single signal as ground truth

WHAT YOU LEARN:
  → Staging buys recall at the cost of money and latency
  → No model mix dominates every axis
  → Severity splits reveal what aggregate scores hide
  → "Best" is meaningless without specifying the tradeoff you care about

---

Quick Reference Summary