Step-by-Step Teaching

Step 1: Why "Retraining" Is a Dangerous Oversimplification

Most people treat model retraining as a single action. This is the root problem.

Think of it like going to a doctor. "Treatment" is not one thing — it could mean medication, surgery, or watchful waiting. Prescribing the wrong one causes harm even with good intentions.

The word "retrain" collapses three fundamentally different decisions into one, which causes teams to default to whichever feels most familiar rather than whichever is actually correct.

---

Step 2: The Three Actions Defined

🔄 Action 1: Refit

What it is: Same model structure, same features — just updated with newer data

Analogy: Recalibrating a scale with fresh weights. The scale itself is fine; it just needs adjustment.

When it applies:

• The world has changed gradually
• Parameters have drifted but the underlying process is the same
• Routine maintenance between major events

Hidden cost: It is the cheapest option, but not free

• Validation still required
• Recent unusual data can degrade a model that was actually fine
• Every refresh is a small risk

---

🔧 Action 2: Respecify

What it is: Change the model itself — add/remove features, change structure, alter assumptions

Analogy: Replacing the engine in a car, not just refueling it. The car's fundamental mechanism needs to change.

When it applies:

• The process generating data has fundamentally changed
• Fresh data cannot fix the problem because the model has no language to describe the new world
• The model misses in the same direction, repeatedly

Key insight: Respecification works in two directions:

The forgetting problem: Teams readily add structure during crises but rarely remove it afterward. This is where "ghosts" are born.

---

✋ Action 3: Hold

What it is: Examine the miss, decide it does not warrant action, leave the model alone

Analogy: A doctor seeing a slightly elevated reading and saying "let's monitor this" rather than immediately prescribing medication.

When it applies:

• The miss falls within the model's already-predicted normal range
• You cannot yet explain the surprise (unexplained surprises are the worst reason to move a model)
• An off-cycle event caused a temporary spike

Why it's hard: It looks like negligence. Saying "we chose not to act" is unpopular when stakeholders are watching a miss accumulate.

The article's maxim: "Don't just do something; sit there."

---

Step 3: The Three Failure Modes

Each wrong action has a corresponding failure pattern. Recognizing these patterns is your diagnostic tool.

❌ Failure 1: Chasing Noise (Wrong Refit)

What happens: A model misses for one period → team refits early → model absorbs the unusual period as the "new normal"

Real example from the article:

Event → Unusual booking spike → Early refit → 
Model treats spike as new baseline → 
Next two forecasts come in too high → 
Error persists until spike ages out of training window

The core mistake: You are rushing to absorb the window you understand least

Diagnostic signal: Miss is noisy and temporary, not directional

---

👻 Failure 2: Carrying Ghosts (Missed Respecify)

What happens: A structural assumption added during a crisis stays switched on long after the crisis ends, hidden behind routine refits that all look clean

Real example from the article:

COVID → Cancellation timing shifts → Model learns new pattern →
Market returns to normal → Model does NOT return to normal →
Routine refits look fine → Bias accumulates quietly for quarters

The core mistake: Refitting keeps the model current but not honest — it never touches the assumptions underneath

Diagnostic signal: Miss is consistently one-directional over multiple periods

---

😱 Failure 3: Respec-as-Panic (Wrong Respecify)

What happens: A genuine but temporary disruption triggers a full model rebuild, producing a more fragile model tuned to a shock that is already fading

Real example from the article:

FX rate swing → Forecast goes off → Instinct: build new model →
Better solution: widen priors → Existing model rides it out →
Avoided fragile model tuned to a temporary regime

The core mistake: Rebuilding under deadline pressure when the existing model just needed more flexibility

Diagnostic signal: Miss is explainable by a known temporary event

---

Step 4: The Decision Framework

Here is the three-question diagnostic, applied in order:

STEP 1: Is the miss directional (same direction repeatedly)?
    │
    ├── NO → Is it within normal model variance?
    │           ├── YES → HOLD (don't refit off-cycle)
    │           └── NO → REFIT (parameters drifted, structure is fine)
    │
    └── YES → Does adding fresh data make it better or worse?
                ├── BETTER → REFIT (parameter problem)
                └── WORSE/UNSTABLE → RESPECIFY (structural problem)

Summary Table

---

Step 5: The COVID Case Study — Applying the Framework

This is the clearest real-world application of the framework.

Before COVID

• Markets were pooled in a hierarchy
• Stable markets "lent" their stability to thinner markets
• This borrowing was invisible because it worked

What COVID broke

• Recovery was uneven — markets that used to move together diverged
• Markets were being pulled toward an average that described none of them

What happened when they tried to refit

Refit attempt → Estimates went UNSTABLE → 
Swinging quarter to quarter → 
Error = 3x pre-shock baseline → 
Would not settle with new data

Applying the diagnostic

The fix (Respecification)

• Old structure: Borrow stability from fixed geographic hierarchy
• New structure: Borrow from markets with similar recovery dynamics today
• Result: Error cut by ~50%, returned to near pre-shock baseline

---

Step 6: Why This Is Hard in Practice

Understanding the framework is easy. Applying it under pressure is not. Here are the three human factors that make it difficult:

---

Step 7: The Core Principle — Learning to Forget

This is the article's deepest insight:

Refitting keeps the model current. It does not keep it honest.

What goes stale is not the numbers — it is the assumptions underneath them. And those are exactly what a refit leaves untouched.

A mature forecasting practice requires two skills:

1. Learning from shocks — adding structure when the world changes
2. Forgetting shocks — removing that structure when the world returns

The second skill is harder because:

• It has no natural deadline
• Nobody schedules it
• Removing feels like losing something
• The bias it prevents is invisible until it has already accumulated

---

Quick Reference Summary

REFIT    → Same model, new data. For gradual drift. Cheapest but not free.
RESPECIFY → New model structure. For fundamental process change. 
            Can ADD or REMOVE assumptions. Removing is the forgotten half.
HOLD     → No action. For noise, temporary events, unexplained surprises.
            Hardest to justify. Often correct.

KEY DIAGNOSTIC: Does fresh data help?
    YES → Parameter problem → Refit
    NO  → Structural problem → Respecify
    
KEY PRINCIPLE: A refit updates numbers. Only a respecify updates assumptions.
               A good team learns from shocks AND learns to forget them.