After studying this material, you should be able to:
- Explain the limitations of standard RLVR on extremely difficult tasks
- Describe how RLTL;DR works as a self-improvement mechanism
- Understand the role of TL;DR insights and internalization
- Compare RLTL;DR vs SFTL;DR vs standard approaches
- Evaluate why task→insight mapping is the key innovation
Step-by-Step Teaching
Step 1: The Problem — Why Standard RL Fails on Hard Tasks
Standard RLVR Paradigm
Agent attempts task → Gets reward signal → Learns from successes
The Core Assumption:
The agent must succeed at least sometimes to learn
The Breaking Point
When tasks are extremely difficult:
| Scenario | Problem |
|---|---|
| Pass@128 = 0% | Agent never succeeds even in 128 tries |
| No teacher model | Nothing to distill from |
| No example solutions | No demonstrations available |
Think of it like this:
Imagine learning to ride a bike, but you never stay balanced even for a second. Standard RL gives you nothing to learn from because there are zero successful moments to reinforce.
Step 2: The Core Idea — Learning From Failure, Not Just Success
RLTL;DR asks:
"What if we could extract useful learning signals even from failed attempts?"
The Key Insight
Even when an agent fails, the verifier knows:
- ✅ What went wrong
- ✅ How close the attempt was
- ✅ What rule was violated
This information is wasted in standard RLVR.
Step 3: How RLTL;DR Works — The Mechanism
The Sequential Process
┌─────────────────────────────────────────────┐
│ ATTEMPT 1 │
│ Task → Agent tries → FAILS │
│ Verifier output shown to agent │
│ Agent writes: "TL;DR: I forgot to X" │
└──────────────────┬──────────────────────────┘
│ Insight 1 stored
┌──────────────────▼──────────────────────────┐
│ ATTEMPT 2 │
│ Task + Insight 1 → Agent tries → FAILS │
│ Agent writes: "TL;DR: Also need to Y" │
└──────────────────┬──────────────────────────┘
│ Insights 1+2 stored
┌──────────────────▼──────────────────────────┐
│ ATTEMPT N │
│ Task + All Insights → Agent SUCCEEDS ✅ │
└─────────────────────────────────────────────┘
What is a "TL;DR Insight"?
- A single sentence summary of what went wrong
- Written by the agent itself (self-generated feedback)
- Acts as a memory aid for the next attempt
Example (coding task):
"TL;DR: I need to handle edge cases where the input list is empty before iterating."
Step 4: The Internalization Step — The Real Magic
Two Levels of Learning
Level 1: Insights In Context (Easy)
Task + [Insight 1, Insight 2...] → Better Answer
- Agent uses insights as external memory
- Works well during training
- Problem: At evaluation time, no insights exist yet!
Level 2: Task → Insight Internalization (Key Innovation)
Backpropagation on (Task → Insight) mapping
The model learns:
"When I see THIS TYPE of task, I should REMEMBER this type of thing"
Analogy:
A student who not only writes notes during practice but memorizes the pattern of mistakes so they don't need notes during the exam.
Why This Matters
Training time: Task + Insights → 14-31% Pass@1 ✅
Eval time: Task alone → 12-13% Pass@1 ✅ (still works!)
Standard GRPO: Task alone → 0-1% Pass@1 ❌
Step 5: SFTL;DR — The Simplified Discovery
What They Found
Researchers asked: "What's the minimum needed to get this benefit?"
SFTL;DR Setup
Train ONLY on:
(Task, Insight) pairs
— No full rollouts
— No RL rewards
— Just 4,000 examples
Surprising Result
| Method | Data Used | Pass@1 |
|---|---|---|
| Standard GRPO | Full rollouts | ~0-1% |
| RLTL;DR | Full rollouts + insights | 12-13% |
| SFTL;DR | Only 4k (task, insight) tuples | ~Similar to RLTL;DR |
| Classical SFT | Full rollouts | ~Similar |
The implication is profound:
You don't need to show the model how to solve the task. You just need to show it what to keep in mind.
Step 6: The Big Picture — A New Training Paradigm
Traditional Paradigm
"Here is a task. Here is the full correct solution. Learn to replicate it."
RLTL;DR Paradigm
"Here is a task. Here is what to keep in mind. Figure out the solution yourself."
Why This Is Powerful
┌─────────────────────────────────────────┐
│ COMPACT KNOWLEDGE REPRESENTATION │
│ │
│ Full solution: ~500-2000 tokens │
│ TL;DR insight: ~10-30 tokens │
│ │
│ Yet performance is nearly equivalent! │
└─────────────────────────────────────────┘
Step 7: Results Summary
Experimental Setup
- Model: Qwen 3.5 9B Thinking policy
- Tasks: Tool-calling + Coding (filtered to Pass@128 = 0)
- Metric: Pass@1 (does it solve on first try?)
Results
Standard GRPO: ████░░░░░░░░░░░░ 0-1%
RLTL;DR (with insights): ████████████████ 14-31%
RLTL;DR (no insights): ███████████████░ 12-13%
SFTL;DR (4k tuples): ██████████████░░ ~12-13%
Key Takeaways
| Concept | Summary |
|---|---|
| Problem | Standard RL fails when success rate is 0% |
| Solution | Extract learning from failures via self-written insights |
| Mechanism | Sequential rollouts conditioned on accumulated TL;DR insights |
| Key Innovation | Backpropagate on task→insight mapping, not just task→solution |
| Surprising Finding | Compact (task, insight) pairs alone nearly match full training |
| Implication | "What to keep in mind" may be more learnable than "how to solve" |
Self-Check Questions
- Why does standard RLVR fail when Pass@128 = 0?
- What is a TL;DR insight and who generates it?
- Why does internalization matter for evaluation time performance?
- What does SFTL;DR prove about the role of full rollouts?
- In your own words, what is the new training paradigm this paper proposes?