SymptomAI: Towards a conversational AI agent for everyday symptom assessment

Concept 1: The Problem with Current Symptom Assessment

What is the issue?

Most clinical diagnoses can be made through language-based interviews — a doctor simply asking you questions about how you feel. However, these interviews have real-world barriers:

• 💰 Financial — doctor visits are expensive
• 🗺️ Geographic — not everyone lives near a clinic
• 🏥 Systemic — scheduling delays, wait times, etc.

Why can't we just use existing AI?

Current AI language models (LMs) have been tested on curated, highly detailed medical case studies — essentially "textbook perfect" patient descriptions. But real patients:

• Use varying levels of medical vocabulary
• Give incomplete information
• Communicate in messy, natural conversation

Key Insight: There is a significant gap between how AI performs on clean test cases versus how it would perform with real everyday patients.

---

Concept 2: What is Differential Diagnosis (DDx)?

Definition

A differential diagnosis (DDx) is a ranked list of possible diagnoses that could explain a patient's symptoms.

Example

If you report fever, sore throat, and fatigue, a DDx might include:

1. Strep throat
2. Influenza
3. Mononucleosis
4. Common cold

Why a list and not one answer?

Because symptoms overlap across many conditions. A clinician — or AI — narrows down possibilities through follow-up questions before arriving at a most likely diagnosis.

Key Metric Used: Top-5 Accuracy — whether the true diagnosis (later confirmed by a real doctor) appears anywhere in the AI's list of 5 candidates.

---

Concept 3: How SymptomAI Was Designed and Tested

The Setup

SymptomAI is a conversational AI agent built on Gemini Flash 2.0 that:

1. Interviews patients about their symptoms
2. Asks follow-up questions
3. Produces a differential diagnosis list
4. Recommends next steps

The Scale

• 13,917 participants enrolled nationally
• Participants consented and described symptoms to the AI
• Two weeks later, they reported what a real doctor diagnosed them with
• This real doctor's diagnosis became the ground truth for evaluation

The Evaluation

A panel of three board-certified clinicians:

• Read the conversation transcripts
• Provided their own DDx
• Then blindly ranked both SymptomAI's DDx and each other's DDx

⚠️ Important Note: All AI-generated diagnoses were for research purposes only — not official medical assessments.

---

Concept 4: The Five Experimental Approaches (Study Arms)

Participants were randomly assigned to one of five agent types, each with a different interview style:

Key Finding

All four agent-driven approaches significantly outperformed the Base condition.

Lesson: When AI actively asks follow-up questions, diagnostic accuracy improves dramatically compared to just letting users type whatever they want.

---

Concept 5: SymptomAI vs. Real Clinicians — The Results

Finding 1: Clinicians Preferred SymptomAI's DDx

In more than 50% of cases, clinicians ranked SymptomAI's DDx as better than those provided by other clinicians.

Finding 2: Higher Top-5 Accuracy

SymptomAI's DDx more frequently contained the true diagnosis (as confirmed by the participant's personal doctor) compared to clinician-generated DDx.

Finding 3: Greatest Advantage in Uncertain Cases

SymptomAI performed best relative to clinicians in cases where clinicians themselves felt least confident — suggesting AI may add the most value precisely where human uncertainty is highest.

Big Picture: SymptomAI performed at or above the level of board-certified clinicians in this research setting.

---

Concept 6: Wearable Biosignals as Validation Evidence

The Idea

If SymptomAI correctly identifies someone as having a respiratory infection, we should be able to see physical evidence of that illness in their body data — even before they reported symptoms.

How It Was Done

• Participants wore Fitbit devices
• Up to 30 days of biometric data was collected before their SymptomAI conversation
• Metrics tracked: heart rate, respiration, skin temperature, sleep quality

What Was Found

Participants diagnosed by SymptomAI with respiratory infections showed:

• Clear physiological shifts in the days leading up to their conversation
• These shifts peaked around Day 0 (the day of the conversation)
• Participants with non-infectious conditions showed no such pattern

Biosignal Change
      ↑
      |          🔴 Infected (respiratory)
      |        ╱
      |      ╱
      |____╱___________________________
      |    ⬛ Baseline (all others)
      |
      ←————————————————————————→
    Day -30    Day -15    Day 0 (SymptomAI conversation)

Why This Matters: The wearable data independently confirms SymptomAI's diagnoses — providing objective, physiological evidence that the AI got it right.

---

Concept 7: Why This Enables Population-Scale Research

The Current Problem

To study health patterns across large populations, researchers need clinical-quality labels (i.e., someone needs to diagnose each person). This is:

• Extremely expensive
• Slow
• Impossible at massive scale

What SymptomAI Enables

By accurately diagnosing thousands of people automatically, SymptomAI can:

• Label large datasets at clinical quality
• Pair diagnoses with wearable data across populations
• Enable discovery of biosignal patterns for many diseases simultaneously

Analogy: It's like having a tireless clinician who can interview and diagnose millions of people, then hand that data to researchers to find patterns no single hospital could ever study.

---

Concept 8: Honest Limitations

The researchers openly acknowledge several important caveats:

Limitation 1: Diagnosis is a Moving Target

• Symptoms evolve over time
• Some participants reported symptoms too early (before clear signs developed)
• Others reported late (with chronic conditions already well understood)

Limitation 2: Clinicians Were Constrained

• Clinicians reviewed static transcripts — they couldn't ask their own follow-up questions
• In real life, clinicians also use: body language, visual cues, medical history, and existing patient relationships

Limitation 3: AI Has Blind Spots

• SymptomAI cannot physically examine a patient
• It may miss signals that only an in-person visit would reveal

---

Summary: The Full Picture

PROBLEM → Symptom assessment has barriers; AI tested only on clean data

SOLUTION → SymptomAI: conversational AI that interviews real patients

METHOD → 13,917 participants, 5 agent types, clinician evaluation panel

RESULT 1 → SymptomAI DDx preferred over clinicians' in 50%+ of cases

RESULT 2 → Wearable biosignals independently validate AI diagnoses

RESULT 3 → Active follow-up questioning is key to accuracy

FUTURE → Population-scale health research now becomes feasible

Core Takeaway: SymptomAI demonstrates that conversational AI can conduct real-world symptom interviews at or above clinician-level accuracy — and when paired with wearable data, opens entirely new possibilities for large-scale health research.