Defined Learning Outcomes
After studying this material, students should be able to:
- Explain how AI safety classifiers work and why they sometimes block legitimate requests
- Distinguish between different types of AI jailbreaks by severity
- Evaluate the tradeoffs between AI capability access and safety measures
- Understand how industry and government coordinate on AI security
Step-by-Step Teaching Guide
STEP 1: Foundation — What Are AI Safety Classifiers?
Think of it like airport security:
All passengers → Security screening → Allowed through OR Stopped
All AI requests → Safety classifier → Allowed response OR Blocked
Key concept: Classifiers are NOT perfect
They make two types of mistakes:
| Mistake Type | What Happens | Example |
|---|---|---|
| False Negative | Harmful request gets through | Dangerous code produced |
| False Positive | Safe request gets blocked | Legitimate security research refused |
Core Principle:
Classifiers deliberately block some safe requests to ensure harmful requests are never missed
STEP 2: Understanding the "Safety Margin"
This is the most important concept in the article.
Visualize it as a spectrum:
SAFE ←————————————————————————→ HARMFUL
[Clearly Safe] [Gray Area] [Safety Margin] [Blocked Zone]
✅ ⚠️ 🚫 🚫
The Safety Margin explained:
- Requests must look very clearly safe to pass through
- Anything ambiguous gets blocked
- This frustrates users but prevents dangerous outputs
Fable 5 specifically made this margin LARGER:
Normal Model: [——Safe——][—Margin—][——Blocked——]
Fable 5: [—Safe—][———Larger Margin———][——Blocked——]
Tradeoff:
- ✅ Fewer harmful outputs escape
- ❌ More legitimate requests get refused
STEP 3: The Jailbreak Severity Framework
What is a jailbreak?
A technique that tricks the AI's safety system into producing outputs it was designed to block
The Five Categories (from least to most severe):
ROW A — Normal Operation
[Safe requests pass] [Safety margin blocks ambiguous] [Harmful blocked]
ROW B — Fable 5 (Extra Large Margin)
[Fewer safe requests pass] [Much larger margin] [Harmful blocked]
ROW C — Minor Jailbreak ⚠️ LOW SEVERITY
User bypasses classifier BUT lands inside safety margin
→ Output is still very unlikely to be harmful
ROW D — Narrow Harmful Jailbreak ⚠️⚠️ MODERATE SEVERITY
User bypasses classifier AND reaches specific harmful behavior
→ Limited damage because it only works for one narrow task
ROW E — Universal Jailbreak ⚠️⚠️⚠️ HIGH SEVERITY
User bypasses classifier AND unlocks MANY harmful behaviors
→ Most dangerous category
Real-world analogy:
| Jailbreak Type | Lock Analogy |
|---|---|
| Minor (C) | Picking a lock but the door leads to an empty room |
| Narrow Harmful (D) | Picking a lock and accessing one specific file |
| Universal (E) | Master key that opens every door in the building |
STEP 4: The Proposed Industry Scoring Framework
Four criteria for measuring jailbreak severity:
Criterion 1: Capability Gain
"Does this jailbreak give attackers something they couldn't already get?"
LOW SCORE: Other free tools can do the same thing
HIGH SCORE: Unlocks capabilities beyond any existing tool
Criterion 2: Breadth of Capability Gain
"How many different attacks does this enable?"
LOW SCORE: Only works for one very specific attack
HIGH SCORE: Works across many different attack types
Criterion 3: Ease of Weaponization
"How hard is it to actually use this jailbreak to cause harm?"
LOW SCORE: Requires many tries, expert knowledge, complex prompting
HIGH SCORE: Works on first attempt with simple prompt
Criterion 4: Discoverability
"How easy is it to find this technique?"
LOW SCORE: Requires specialist knowledge to discover
HIGH SCORE: Already posted publicly online
Scoring Summary Table:
| Criterion | Low Risk | High Risk |
|---|---|---|
| Capability Gain | Same as existing tools | Far beyond existing tools |
| Breadth | One narrow task | Many attack types |
| Weaponization | Hard, many retries needed | Easy, works immediately |
| Discoverability | Specialist knowledge required | Publicly available |
STEP 5: The Real-World Case Study (Fable 5 Incident)
Apply your knowledge to what actually happened:
Timeline:
June 9 → Fable 5 & Mythos 5 released
June 12 → Export controls applied (access suspended for ALL users)
June 26 → Mythos 5 access restored for US organizations
June 30 → Export controls lifted
July 1 → Fable 5 restored globally
What triggered the export controls?
- Amazon researchers found a bypass technique
- It allowed Fable 5 to identify software vulnerabilities
- In one case, it produced code showing how to exploit a vulnerability
How was it assessed using the framework?
| Criterion | Assessment | Score |
|---|---|---|
| Capability Gain | LOW — Other models (GPT-5.5, Opus 4.8) could do the same thing | Low |
| Breadth | LOW — Narrow, specific behavior | Low |
| Weaponization | Not specified as easy | Moderate |
| Discoverability | Known to researchers | Moderate |
Conclusion: This was a minor jailbreak (Row C) — it breached the safety margin but did not unlock genuinely dangerous unique capabilities
The Fix:
Problem identified → New classifier trained → Blocks technique in 99%+ of cases
STEP 6: Government & Industry Collaboration Model
Why does this matter?
Without coordination:
Jailbreak discovered → Confusion about severity → Inconsistent response → Potential harm
With a shared framework:
Jailbreak discovered → Score it on 4 criteria → Calibrated response → Consistent communication
Four pillars of government collaboration:
1. PRE-RELEASE ACCESS
Government tests models BEFORE public release
2. RAPID INFORMATION SHARING
Jailbreaks reported quickly to government partners
3. JOINT RESEARCH
Shared teams, computing resources, expertise
4. COMMON INDUSTRY STANDARD
Voluntary security standards applied equally to all AI developers
STEP 7: Key Tensions & Tradeoffs to Remember
The fundamental tension in AI safety:
CAPABILITY ←————————————→ SAFETY
More capable models vs. Safer, more restricted models
More user access vs. Tighter controls
Faster deployment vs. More thorough testing
Defense in Depth principle:
No single safety measure is perfect. Layer multiple imperfect defenses so that bypassing one layer doesn't compromise everything.
Layer 1: Model training (refuses dangerous requests)
Layer 2: Safety classifiers (blocks harmful outputs)
Layer 3: Usage pattern monitoring (detects misuse after the fact)
Layer 4: Government coordination (external oversight)
Quick Review: Key Terms
| Term | Simple Definition |
|---|---|
| Safety Classifier | Automated AI system that screens requests for potential harm |
| Safety Margin | Buffer zone where borderline-safe requests are blocked as precaution |
| Jailbreak | Technique that tricks safety systems into producing blocked outputs |
| False Positive | Safe request incorrectly blocked |
| False Negative | Harmful request incorrectly allowed through |
| Defense in Depth | Using multiple overlapping safety layers |
| Universal Jailbreak | Technique that bypasses safety across many harmful behaviors |
| Red-teaming | Deliberately trying to break a system to find vulnerabilities |
Self-Check Questions
- Why does a larger safety margin increase false positives?
- What makes a universal jailbreak more dangerous than a narrow harmful jailbreak?
- In the Fable 5 incident, why was the jailbreak considered low severity?
- Why is a common industry framework for jailbreaks valuable?
- What is the purpose of defense in depth in AI safety?