Self-Distillation vs. GRPO: Learning Without Forgetting

Defined Learning Outcomes

After studying this material, you should be able to:

1. Explain the core problem of catastrophic forgetting in continual LLM post-training
2. Describe how Self-Distillation and GRPO work as on-policy training methods
3. Compare the trade-offs between Self-Distillation and GRPO across skill acquisition and retention
4. Identify failure modes specific to Self-Distillation
5. Apply insights to make informed decisions about which method to use in production settings

---

Step 1: Understand the Core Problem

What is Catastrophic Forgetting?

When you train a model on a new skill, it tends to overwrite what it already knew. This is called catastrophic forgetting.

Base Model (knows math, code, logic)
        ↓ train on Tool-Use
Updated Model (better at Tool-Use, worse at math/code/logic)

Why This Matters in Production

At DoorDash, models are never static. New tools and tasks are constantly added. Each training round must:

• ✅ Add new capabilities
• ✅ Preserve existing capabilities

Key Insight: Standard Supervised Fine-Tuning (SFT) is cheap but causes significant forgetting. On-policy RL methods tend to preserve existing capabilities better.

---

Step 2: Understand the Two Methods Being Compared

Method A: GRPO (Group Relative Policy Optimization)

How it works:

1. Model generates multiple responses to the same prompt
2. Responses are graded by a reward signal (correct/incorrect)
3. Model updates toward responses that scored higher

Prompt → [Response 1, Response 2, Response 3, Response 4]
                ↓ grade each
         [Wrong,  Right,  Wrong,  Right]
                ↓ learn
         Reinforce patterns from correct responses

Key properties:

• Learns from its own attempts
• Requires a reward signal (verifiable answers)
• Requires 5x+ more compute than SFT
• Tends to generalize rather than memorize

---

Method B: Self-Distillation

How it works:

1. A student model sees only the question
2. A teacher copy of the same model sees the question plus a worked demonstration
3. Training minimizes the difference (KL divergence) between their token-by-token predictions along the student's own response

Student: [Question] → generates response
Teacher: [Question + Demonstration] → generates response

Training signal: Make student's token predictions match teacher's

Key properties:

• Combines learning from demonstrations with on-policy training
• Does not require a reward signal
• More compute-efficient than GRPO
• Aims to get the retention benefits of RL with the guidance of SFT

---

Step 3: Understand the Experiment Design

What Was Tested

Training was sequential: each stage started from the previous stage's weights.

What Was Measured

Five independent themes tested retention of pre-existing capabilities:

---

Step 4: Understand the Results

4A: Skill Acquisition

Critical finding: Self-Distillation actually fell below the base model on Knights & Knaves (47.7% → 31.6%). GRPO nearly mastered it (95.9%).

---

4B: Retention of Existing Capabilities

Self-Distillation losses after all 3 stages:

Logic:                -30 points  ❌
Instruction Following: -21 points  ❌
Math:                 -15 points  ❌
Code:                  -5 points  ❌

GRPO changes after all 3 stages:

Math:                 +7 points   ✅
Logic:                +4 points   ✅
Instruction Following: +3 points  ✅
No theme dropped more than 0.2 points ✅

Key Insight: GRPO not only avoided forgetting — it improved on themes it was never trained on.

---

4C: Impact on Reasoning Quality

Answer length (AIME 2025 math problems):

Self-checking behavior (AIME 2025):

Key Insight: GRPO learned to verify its own work. Self-Distillation learned to produce shorter answers with less reflection.

---

Step 5: Understand Why Self-Distillation Fails

This is the most important conceptual section. Self-Distillation has a structural flaw in how it learns.

The Core Problem

The teacher has seen the answer. The student learns to write as if it had also seen the answer — even though it hasn't.

Failure Mode 1: Hallucinating the Teacher's Example

Student's reasoning: "Since the example arrives at 270, 
I will follow its logic to ensure consistency."

The student never saw the example. It hallucinates that it did, and defers to a non-existent authority instead of its own calculation.

Failure Mode 2: Format Contamination

After training on Knights & Knaves last, the model applies that task's answer format to all other tasks — even math and instruction-following.

Training order: Tool-Use → Science → Knights & Knaves
Result: Math answers now formatted like logic puzzle answers

Failure Mode 3: Instruction Inversion

Asked to rewrite a passage starting with a verb:

Self-Distillation output:
"The response must start with a verb, so I will begin with 'Cultivate'..."

It describes the rule in a sentence that breaks the rule.

Failure Mode 4: Reasoning Collapse (with Thinking Mode)

With thinking enabled, the model's reasoning traces shrink dramatically:

Base Model (thinking on): 10,092 tokens median
After Self-Distillation:   4,768 tokens median

The model stops "searching" for answers because the teacher already knew them.

---

Step 6: Understand the Nuances

What Happens When You Train on All Skills Together?

Instead of sequential training (Tool-Use → Science → K&K), train on all three shuffled together:

But: Answer length stays short (median 495 tokens) and self-checking stays at 2%.

Conclusion: Mixing softens forgetting but does not fix the reasoning quality problem.

---

What About Knowledge?

Knowledge benchmarks (MMLU, TruthfulQA, etc.) barely moved under either method (< 4 points change).

Interpretation: Facts are stored differently than procedures. Post-training on reasoning tasks affects how the model reasons, not what facts it knows.

---

Step 7: Synthesize the Key Trade-offs

---

Summary: The Mental Model

Self-Distillation learns:
"Here's what someone who already knew the answer would write"
→ Shortcuts reasoning
→ Borrows authority it doesn't have
→ Forgets how to follow instructions independently

GRPO learns:
"Here's what actually worked when I tried it myself"
→ Deepens reasoning
→ Learns to verify
→ Stays close to original behavior

---

Self-Test Questions

1. Why does on-policy training generally cause less catastrophic forgetting than SFT?
2. What is the structural reason Self-Distillation causes the model to hallucinate "the example response"?
3. Why did GRPO improve on math even though it was never trained on math benchmarks?
4. What does the answer length data tell us about reasoning quality?
5. Under what conditions might Self-Distillation be preferable to GRPO in a production setting?
6. Why does training on mixed data reduce forgetting in Self-Distillation but not fix reasoning collapse?