Turning Failed Teacher Trajectories Into Better Tool Agents

🎯 Defined Learning Outcomes

After studying this material, you should be able to:

1. Explain the standard knowledge distillation pipeline and its core limitation
2. Describe what PROOF-Gen is and the problem it solves
3. Understand the mechanism of per-scenario reflective optimization
4. Interpret the empirical results and what they mean practically
5. Connect PROOF-Gen to broader distillation concepts (on-policy distillation, scaling laws)

---

📚 Step-by-Step Teaching

---

STEP 1: Foundation — What is Knowledge Distillation?

Core Concept: Knowledge distillation is the process of transferring capabilities from a large, expensive "teacher" model into a smaller, deployable "student" model.

Teacher Model          Student Model
(Large, Expensive)  →  (Small, Fast, Deployable)
GPT-4 class            Phone/Edge device class

How it works in tool-calling specifically:

"Tool-calling" means a model can invoke external tools (APIs, calculators, search engines) to complete tasks.

The standard pipeline:

1. Teacher model attempts tasks
2. Teacher generates "trajectories" (step-by-step tool-call sequences)
3. Passing trajectories → kept as training data
4. Student model learns by imitating these clean demonstrations

Key Term: Supervised Fine-Tuning (SFT) — training a model by showing it correct input-output examples repeatedly.

---

STEP 2: Identifying the Core Problem

The Generate-and-Filter Problem:

The standard pipeline has a critical flaw:

Teacher attempts task
        ↓
    PASS? ──YES──→ Keep trajectory → Train student
        │
       NO
        ↓
    DISCARD ← ❌ No learning signal extracted

Why this is wasteful — by the numbers:

Near-miss = most tool calls were correct, but one decisive error ruined the whole trajectory

The compounding problem:

• Post-training pipelines run daily or weekly
• Each cycle pays the expensive teacher cost again
• Each cycle fails on the same hard scenarios because failures provide zero corrective signal
• Hard problems stay hard — forever

Analogy:

Imagine a student who takes a test, gets 9/10 questions right but fails because of question 7. The teacher throws away the entire test paper and gives the same test again next week. The student never learns why question 7 was wrong.

---

STEP 3: The PROOF-Gen Solution

PROOF-Gen = Per-scenario Reflective Optimization to Overcome Failed Generation

Core Insight:

Instead of discarding failures, analyze them and guide the teacher to succeed on the same scenario.

The PROOF-Gen Pipeline:

Teacher FAILS on Task X
         ↓
    REFLECTOR MODULE
    ├── Analyzes execution trace (what happened step by step)
    ├── Reads evaluation feedback (why it failed)
    └── Writes "corrective guidance" (specific instructions to fix the error)
         ↓
Teacher re-attempts Task X WITH corrective guidance
         ↓
    Teacher PASSES ✓
         ↓
Guidance is STRIPPED OUT
         ↓
Clean trajectory stored → Student trains on it

Critical Design Choice — Why strip the guidance?

The student sees only the clean, correct trajectory — not the hints that helped the teacher get there. This is like a student learning from a polished worked example, not from the teacher's rough draft notes.

---

STEP 4: Understanding the Results

Recovery Rate:

Failed scenarios before PROOF-Gen:  57% of all tasks
Recovered by PROOF-Gen:             93% of those failures

This means PROOF-Gen converts most "wasted" failures into usable training data.

Student Model Performance Gains:

Pass@1 = probability the model solves a task correctly on the first attempt

Real-World Deployment Results:

Key observation: Positive transfer in every locale — the improvement is not limited to English, suggesting the method improves fundamental reasoning, not surface-level pattern matching.

---

STEP 5: Connecting to Broader Distillation Research

Two related research directions mentioned:

5a. On-Policy Distillation

• What it is: The teacher provides supervision signal token by token during student training (dense supervision)
• Open question: When does this help vs. hurt?
• Relevance to PROOF-Gen: PROOF-Gen operates at the trajectory level (whole sequences), while on-policy distillation operates at the token level — complementary approaches

5b. Distillation Scaling Laws

• What it is: Mathematical relationships predicting how student performance scales with compute budget
• Key insight: Optimal compute should be split between teacher and student
• Relevance to PROOF-Gen: PROOF-Gen changes the quality of teacher data, while scaling laws govern quantity of compute — together they address both dimensions of distillation efficiency

---

STEP 6: Conceptual Summary Map

PROBLEM
└── Standard distillation wastes 57% of teacher compute
    └── Hard scenarios never improve cycle-to-cycle

SOLUTION: PROOF-Gen
├── Reflector analyzes failures
├── Writes per-scenario corrective guidance
├── Teacher re-attempts → succeeds
└── Guidance stripped → clean data for student

RESULTS
├── 93% failure recovery
├── Up to 4x improvement in Pass@1
└── Positive transfer across languages & deployment contexts

BROADER CONTEXT
├── On-policy distillation → token-level supervision questions
└── Scaling laws → compute-optimal teacher/student allocation

---

✅ Self-Check Questions

1. Why does the standard generate-and-filter pipeline fail to improve on hard scenarios over time?
2. What does the "reflector" module do in PROOF-Gen, and why is its output stripped before training?
3. What does a 93% recovery rate mean in practical terms?
4. Why is positive transfer across non-English locales significant?
5. How does PROOF-Gen differ from on-policy distillation in its approach to supervision?

---

💡 Key Takeaway

PROOF-Gen reframes failures as opportunities. Instead of discarding expensive teacher failures, it extracts corrective signal, recovers clean trajectories, and trains better students — all without leaking task-specific hints into the training data. The result is a more data-efficient distillation pipeline that compounds improvements rather than repeating the same mistakes.