How to Evaluate Patient-Facing Health AI Agents

Concept 1: The Evolution of Health AI — From Q&A to Agentic Systems

What's the basic idea?

Health AI didn't start where it is today. It evolved through distinct stages:

Stage 1: Question Answering
"What are the symptoms of diabetes?"
→ AI gives a static answer

Stage 2: Clinical Decision Support
"Given these lab results, what's the diagnosis?"
→ AI assists a doctor with a specific task

Stage 3: Agentic Systems (Current Frontier)
"Help this patient manage their care"
→ AI converses, reasons over records, AND takes actions

Why does this distinction matter?

An agent is fundamentally different from a chatbot because it:

• Converses over multiple turns (not just one question)
• Reasons over real health records
• Acts on behalf of the patient (scheduling, referrals, triage)

🔑 Key Insight: When AI moves from answering to acting, the stakes change dramatically. A wrong answer is bad. A wrong action — like scheduling a routine appointment when someone needs emergency care — can be dangerous.

---

Concept 2: The Problem with Existing Benchmarks

What are current benchmarks doing?

Most medical AI benchmarks look like this:

Example: MedQA / USMLE-style benchmark

Question: "A 45-year-old presents with chest pain, 
           diaphoresis, and shortness of breath. 
           What is the most likely diagnosis?"

(A) Panic attack
(B) Myocardial infarction  ✓
(C) GERD
(D) Pulmonary embolism

What's wrong with this approach for agentic systems?

The core gap:

🔑 Key Insight: Testing whether an AI knows medicine is not the same as testing whether it can safely act as a patient's healthcare agent. A student can ace a medical exam and still be a dangerous doctor.

---

Concept 3: The PatientAgentBench Architecture

How is the benchmark structured?

Think of it as a testing environment with three interacting components:

┌─────────────────────────────────────────────┐
│           PatientAgentBench Setup           │
│                                             │
│  ┌──────────────┐      ┌──────────────────┐ │
│  │  Foundation  │      │  Healthcare Tool │ │
│  │    Model     │ ←──→ │    Sandbox       │ │
│  │  (the agent) │      │ (schedule, triage│ │
│  └──────┬───────┘      │  lookup records) │ │
│         │              └──────────────────┘ │
│         │ converses                         │
│         ↓                                   │
│  ┌──────────────┐                           │
│  │  Simulated   │                           │
│  │   Patient    │                           │
│  └──────────────┘                           │
└─────────────────────────────────────────────┘

Breaking down each component:

1. The Foundation Model (wrapped as an agent)

• This is the AI being tested (e.g., GPT-4, Claude, Gemini)
• It's given tools and a patient-facing role
• It must decide when and how to use those tools

2. The Healthcare Tool Sandbox

• A controlled set of realistic tools the agent can use
• Examples: look up patient records, schedule appointments, flag for triage
• "Sandbox" = safe, simulated environment (no real patients harmed)

3. The Simulated Patient

• Another AI playing the role of a patient
• Has a realistic health record and scenario
• Presents symptoms, asks questions, responds naturally

🔑 Key Insight: By simulating the entire interaction — not just a single question — the benchmark captures failures that only emerge over time, like an agent that gives correct advice early but then forgets to follow up on a red flag.

---

Concept 4: The 1,200 Scenarios

What is a "scenario"?

Each scenario is one complete patient encounter:

Scenario Example (simplified):

Patient Profile:
- 67-year-old with hypertension, Type 2 diabetes
- Current medications: metformin, lisinopril
- Reason for contact: "I've been having headaches"

The agent must:
1. Converse with this patient
2. Use tools appropriately
3. Triage correctly (is this a headache or a hypertensive crisis?)
4. Take correct actions

Why 1,200 scenarios?

• Scale ensures statistical reliability
• Variety covers different conditions, demographics, urgency levels
• Same scenarios used for all 10 models → fair comparison

🔑 Key Insight: Using the same 1,200 scenarios across all models is critical. It's like giving every student the same exam — otherwise you can't compare scores.

---

Concept 5: LLM-as-a-Jury Evaluation

The evaluation problem

How do you score a complex, multi-turn healthcare conversation? You can't just check if the answer matches a key.

The solution: LLM-as-a-Jury

After each conversation:

Conversation transcript
        ↓
   LLM Jury Panel
        ↓
Scores across 6 dimensions
using 100+ criteria

The 6 Evaluation Dimensions

Think of these as the six things a good patient-facing health agent must do:

What are "conversation-agnostic criteria"?

The 100+ criteria are designed to apply to any conversation, not just specific ones:

Example criteria (illustrative):
✓ "If a patient reports chest pain, agent should assess 
   urgency before proceeding with administrative tasks"
   
✓ "Agent should not claim to have completed an action 
   it did not execute via tools"
   
✓ "In a crisis situation, agent must provide emergency 
   resources"

🔑 Key Insight: Having 100+ criteria means the evaluation is comprehensive — it catches subtle failures, not just obvious ones. "Conversation-agnostic" means the criteria are principled rules, not just checking if the AI matched a specific expected answer.

---

Concept 6: Clinician Validation — Proving the Jury Works

The critical question

If an LLM is judging other LLMs, how do we know the jury is trustworthy?

The validation method

Step 1: Select shared conversations
        (same transcripts given to both jury and humans)
        
Step 2: Licensed clinicians annotate those conversations
        (independent expert ratings)
        
Step 3: Compare jury scores vs. clinician scores

Step 4: Measure agreement

The results

Agreement Rate:
Jury vs. Clinicians: 79–93% adjacent agreement
Clinician vs. Clinician: similar or lower

"Adjacent agreement" = scores within 1 point of each other

Why "adjacent agreement" matters

Scoring systems aren't binary. If a clinician rates something 3/5 and the jury rates it 4/5, that's close enough to be meaningful:

Perfect agreement:  Jury = 4,  Clinician = 4  ✓
Adjacent agreement: Jury = 4,  Clinician = 3  ✓ (within 1)
Disagreement:       Jury = 4,  Clinician = 1  ✗

🔑 Key Insight: The jury is as reliable as human experts — and humans don't always agree with each other either. This validates that the benchmark is measuring something real, not just AI-generated noise.

---

Concept 7: The Benchmark Results — What Was Found

The 10 models tested

Four model families were tested (e.g., GPT, Claude, Gemini, Llama-style), 10 models total, all on the same 1,200 scenarios.

Finding 1: Triage is the most discriminating dimension

Triage Pass Rates:

Weakest models:   ████░░░░░░  32%
                  
Middle models:    ██████░░░░  ~60%

Strongest models: █████████░  88%

What does "triage failure" look like?

Patient: "I've been having crushing chest pain for 20 minutes"
Bad Agent: "I can help you schedule an appointment. 
            What day works for you?"
            
← This is a triage failure: treating an emergency 
  as an administrative task

Finding 2: Clinical Safety and Workflow Accuracy follow the same pattern

Failure types by model strength:

Weak models:
- Fabricate unexecuted actions ("I've scheduled your appointment" 
  → but never used the scheduling tool)
- Fail to screen clinically before acting

Strong models (1–3% failure rate):
- Trust tool outputs without verification
- Omit crisis resources in emergencies

Finding 3: No model is perfect

Best overall score: 4.25 / 5.0

Even the strongest model fails on real cases.

🔑 Key Insight: More capable models are better but not safe. The failures shift from obvious (wrong triage) to subtle (missing one crisis resource) — but they don't disappear.

---

Concept 8: Why Static Benchmarks Are Insufficient

The core argument

The paper makes a strong claim: you cannot evaluate agentic health AI with static benchmarks.

Here's why, illustrated:

Static Benchmark Scenario:
"Patient has BP 180/120 and headache. What do you do?"
→ Model answers: "Refer to emergency care" ✓
→ Benchmark says: PASS

Real Agentic Scenario (what PatientAgentBench tests):
Turn 1: Patient mentions headache
Turn 2: Agent asks about other symptoms
Turn 3: Patient mentions BP reading
Turn 4: Agent... schedules a routine appointment anyway ✗
         (because it got distracted by an administrative 
          request in between)

The failures that only emerge in sustained conversations:

🔑 Key Insight: Agentic failures are emergent — they arise from the interaction of conversation, tools, and time. You can only see them by running the full interaction.

---

Concept 9: The Broader Contribution — A Reproducible Standard

What the paper releases

PatientAgentBench isn't just a study — it's a framework:

Released Components:
├── Evaluation criteria (100+ clinician-grounded)
├── Simulated patient system
├── Scoring methodology (LLM-as-a-Jury)
├── Validation data (clinician annotations)
└── Benchmark scenarios (1,200)

Why reproducibility matters in AI safety

Without a standard:          With PatientAgentBench:
"Our model is safe"          "Our model scores 4.1/5 on 
(no way to verify)           PatientAgentBench, with 85% 
                             triage pass rate"
                             (verifiable, comparable)

🔑 Key Insight: A benchmark only improves the field if others can use it. By releasing the framework, the authors enable every future health AI system to be evaluated against the same bar — creating accountability.

---

Summary: The Full Picture

PROBLEM:
Health AI is becoming agentic (acts, not just answers)
↓
Existing benchmarks only test knowledge, not action
↓
SOLUTION: PatientAgentBench

HOW IT WORKS:
Foundation Model + Tools ←→ Simulated Patient
         ↓
   1,200 scenarios
         ↓
LLM Jury scores 6 dimensions via 100+ criteria
         ↓
Validated against licensed clinicians (79-93% agreement)

WHAT IT FOUND:
• Triage: 32% → 88% pass rate (weakest to strongest)
• All models fail; strong models fail subtly
• Best score: 4.25/5 — no model is safe enough yet
• Failures only visible in sustained, tool-using conversations

CONCLUSION:
Static benchmarks are insufficient.
Agentic health AI needs agentic evaluation.

---

The One Sentence to Remember

PatientAgentBench reveals that even the best AI health agents have dangerous gaps — gaps that are invisible to traditional benchmarks but emerge clearly when you make the AI actually do the job rather than just answer questions about it.