Step-by-Step Teaching

Step 1: Why Clinical Speech Recognition Is Uniquely Hard

Clinical audio is not ordinary conversation. It presents three specific challenges:

Key insight: A general-purpose model trained on everyday speech will struggle here because medical vocabulary is rare, specialized, and unforgiving of mistakes.

---

Step 2: Understanding Word Error Rate (WER)

Before reading any benchmark, you need to understand the core metric.

WER = the percentage of words transcribed incorrectly

$WER = \frac{\text{Substitutions + Deletions + Insertions}}{\text{Total Words in Reference}} \times 100$

Simple Example:

• Actual speech: "Give 10mg of hydroxyzine"
• Transcription: "Give 10mg of hydralazine"
• One word wrong out of five = 20% WER

Rule of thumb:

Lower WER = Better accuracy
Higher WER = More errors

---

Step 3: What Is Model Fine-Tuning?

A base model is trained on broad, general data — everyday conversations, books, podcasts.

Fine-tuning takes that base model and trains it further on a specific domain — in this case, clinical audio.

General Data → Base Model (Scribe v2)
                      ↓
         + Clinical Audio Training
                      ↓
         Fine-Tuned Model (Scribe v2 Medical)

Why does this matter?

• The base model already understands language structure
• Fine-tuning teaches it the specialized vocabulary without starting from scratch
• Result: Better accuracy on medical terms without losing general performance

---

Step 4: Reading the Benchmark Data

There are four benchmarks presented. Let's learn how to read each one.

Benchmark 1 — MedDictate (Clinical Dictation)

Tests transcription of dictated clinical notes in English, French, and German

What to notice:

• Scribe v2 Medical achieves ~35% lower WER than its base version overall
• Deepgram's medical model only supports English — a practical limitation
• Fine-tuning improved German performance the most (9.6% → 7.2%)

---

Benchmark 2 — MedTerm (Medical Terminology)

This benchmark introduces two new metrics beyond basic WER:

What to notice:

• The gap between Scribe v2 Medical and competitors is larger on medical terms than on general speech
• This confirms fine-tuning specifically improved medical vocabulary recognition

---

Benchmark 3 — Omi Health Leaderboard

An independent benchmark — important because it is not run by ElevenLabs

• 1,513 clinical English audio clips
• 57 real consultations
• 30 models tested

Key insight: Independent benchmarks matter. When a company's own model wins on its own tests, that could be biased. Winning on an independent leaderboard is stronger evidence.

---

Benchmark 4 — Eka Medical ASR

Introduces two more metrics:

Granular finding from Eka:

Medical terms in full sentences:  7.5% WER  ✅ (good)
Isolated single drug names:      14.3% WER  ⚠️ (harder)

Why are isolated words harder? Without surrounding context, the model has no linguistic clues to resolve ambiguity.

Memorable failure examples:

---

Step 5: The Solution to Isolated Word Errors — Keyterm Prompting

When you know specific drug names will appear, you can tell the model in advance.

Standard request:
→ Transcribe this audio

Keyterm prompting:
→ Transcribe this audio
→ Expected terms: [etodolac, levomilnacipran, methdilazine]

This biases the model toward recognizing those specific terms, reducing errors on rare or unusual drug names.

---

Step 6: Does Fine-Tuning Hurt General Performance?

A critical question: Does specializing a model make it worse at everything else?

Tested on 6,000 samples of everyday non-medical speech:

Answer: No. The fine-tuned model performs identically on general speech.

This is the ideal outcome of fine-tuning — gain specialization without losing generalization

---

Step 7: Healthcare Compliance — HIPAA and Zero Retention Mode

In healthcare, data privacy is legally required, not optional.

Key concepts:

HIPAA (Health Insurance Portability and Accountability Act)

• US law protecting patient health information
• Any tool handling patient data must be HIPAA-compliant

Business Associate Agreement (BAA)

• A legal contract between a healthcare provider and a vendor
• Required before sharing protected health information (PHI) with a third party

Zero Retention Mode (ZRM)

Audio sent → Transcription processed → Audio deleted immediately
                                     → Transcript deleted immediately
                                     → Only API response returned to you

The vendor retains nothing. Your application controls all data storage.

---

Summary: Key Concepts at a Glance

Clinical Speech Recognition
├── Why it's hard
│   ├── Similar drug names
│   ├── Dense vocabulary
│   └── High stakes
│
├── How performance is measured
│   ├── WER (Word Error Rate) — lower is better
│   ├── Term Recall — higher is better
│   ├── Term-WER — lower is better
│   ├── semWER — semantic equivalence
│   └── kwWER — keyword-specific errors
│
├── Fine-tuning
│   ├── Base model + domain-specific training
│   ├── Improves specialized accuracy
│   └── Does not degrade general performance
│
├── Remaining challenge
│   ├── Isolated single words are hardest
│   └── Solution: Keyterm prompting
│
└── Compliance
    ├── HIPAA eligibility
    ├── Business Associate Agreements
    └── Zero Retention Mode

---

Quick Self-Check Questions

1. Why is clinical audio harder than everyday speech for AI models?
2. If a model has 10% WER, what does that mean in plain language?
3. What is the difference between Term Recall and Term-WER?
4. Why does winning an independent benchmark matter more than winning an internal one?
5. Why are isolated drug names harder to transcribe than drug names inside sentences?
6. What problem does Zero Retention Mode solve in a healthcare context?