After studying this material, you should be able to:

1. Explain what "goodput" is and why it matters for GPU training efficiency
2. Identify the two critical subsystems (checkpointing + data pipeline) that determine training resilience
3. Compare checkpointing strategies (torch.save vs. DCP vs. async_save) and their trade-offs
4. Calculate how checkpoint frequency affects expected wasted work and goodput
5. Describe how to build an overlapping, fault-tolerant data pipeline
6. Recognize the silent failure of not checkpointing data pipeline state

---

Step-by-Step Study Material

Step 1: The Core Problem — What is "Goodput"?

Definition

Goodput = the proportion of time your GPUs spend on productive computation rather than waiting or recovering from failures

Think of it like a factory assembly line:

• A machine that runs 100% of the time = 100% goodput
• A machine that sits idle half the time = 50% goodput

Why Failures Are Inevitable at Scale

Use this mental model:

Each GPU has ~1% annualized failure rate

256-GPU job, 30 days → 19% chance of failure
1,024-GPU job, 30 days → 57% chance of failure

Real-world example: The 608 H100 "Delta" supercomputer saw failures every 1.9 hours. For a 32-GPU job, that means a failure roughly every 36 hours.

Key Takeaway

Your training job will fail. The question is: how much work do you lose when it does?

---

Step 2: Checkpointing — Your First Line of Defense

What is a Checkpoint?

A snapshot of your training state saved to storage so you can resume from that point after a failure, rather than starting over.

The Cost of Infrequent Checkpointing

Expected wasted work per failure ≈ ½ × checkpoint interval

Checkpoint every 24 hours → lose ~12 hours of work on average
Checkpoint every 30 minutes → lose ~15 minutes of work on average

---

Step 3: Three Checkpointing Strategies (Worst → Best)

Strategy 1: torch.save on Rank 0 ❌ (Avoid)

What happens:
[Rank 0] → Gathers ALL model state → Writes entire file → Done
[Rank 1] → Waits...
[Rank 2] → Waits...
[Rank N] → Waits...

Problems:

• Serial bottleneck — only one GPU does the work
• All other GPUs sit idle during the save
• Does not scale — larger models = longer idle time

---

Strategy 2: PyTorch Distributed Checkpoint (DCP) ✅ (Better)

What happens:
[Rank 0] → Writes its own shard ─┐
[Rank 1] → Writes its own shard  ├─ All in PARALLEL
[Rank 2] → Writes its own shard  │
[Rank N] → Writes its own shard ─┘
                                  └→ .metadata file written last

Advantages:

The .metadata file is critical: It is written only after all shards complete. Its presence = trustworthy signal that the checkpoint is valid.

---

Strategy 3: async_save ✅✅ (Best)

Even DCP blocks training until bytes are written to storage. async_save solves this:

Standard DCP:
[Training] → STOP → [Write to storage] → RESUME training

async_save:
[Training] → Fast copy to staging buffer → CONTINUE training immediately
                        ↓
             [Background upload to storage] ← happens while GPU computes

Real performance gains:

The training loop pays only for the staging copy, not the upload.

---

Step 4: How Checkpoint Frequency Compounds Into Goodput

Now connect the pieces with a concrete example:

Scenario: Llama 3 training with ~8.6 interruptions per day

Checkpoint every 2 hours:
  Expected waste per failure = 1 hour
  Total daily waste = 8.6 × 1 hour = 8.6 hours
  Goodput = (24 - 8.6) / 24 = 64%

Checkpoint every 30 minutes:
  Expected waste per failure = 15 minutes
  Total daily waste = 8.6 × 0.25 hours = 2.15 hours
  Goodput = (24 - 2.15) / 24 = 91%

The math is simple:

Cutting checkpoint interval by 10× → cuts expected recovery time by 10×

async_save makes frequent checkpointing affordable because the GPU barely pauses.

---

Step 5: Automatic Recovery

Frequent checkpoints only help if recovery is automatic (no human intervention):

Job fails
    ↓
On restart: scan checkpoint directory
    ↓
Find most recent checkpoint WHERE .metadata file exists
    ↓ (skip any half-written checkpoints from the crash)
Load state → Resume training

The .metadata file acts as a commit marker — if it's there, the checkpoint is safe to use.

---

Step 6: The Data Pipeline Problem

Even with perfect checkpointing, your GPUs can still sit idle — starved by a slow data pipeline.

The Problem: Sequential Data Loading

❌ Bad pattern:
[Load batch] → [GPU computes] → [Load batch] → [GPU computes]
      ↑                               ↑
   GPU idle                        GPU idle

The Fix: Overlapping Data Loading with Computation

✅ Good pattern:
[Load batch N+1] overlaps with [GPU computes on batch N]
[Load batch N+2] overlaps with [GPU computes on batch N+1]

Impact: Customers switching to overlapping data loading see 20–50% decrease in wall-clock time.

---

Step 7: Remote Storage Makes Data Loading Harder

On governed platforms, data lives in remote object storage (e.g., Unity Catalog volumes = network mounts).

Problem with Naive Approach

Every batch access → Network request → Download file → Use it
                          ↑
                    Slow! Repeated every epoch!

Solution: Local NVMe Caching

Epoch 1, first access:
  File → Download from network → Cache to local NVMe → Use

Epoch 1+, subsequent access:
  File → Read from local NVMe (fast!) → Use

Meanwhile: Pre-fetch upcoming files in background

Performance comparison (image classification, same GPU/model/batch):

Epoch 2 is dramatically faster because the cache is warm — files are already on local NVMe.

Monitoring Your Data Pipeline

Track the metric fetch_seconds — this measures how long the dataloader takes to produce a batch. During this time, your GPU is idle.

---

Step 8: The Silent Failure — Not Checkpointing Data Position

This is the most dangerous failure because it produces no error.

What Happens Without Data Position Checkpointing

Training runs → Failure at step 5,000 (mid-epoch)
    ↓
Restore checkpoint: model ✓, optimizer ✓, step ✓
    ↓
Dataloader: starts from BEGINNING of dataset ✗
    ↓
Result: Re-trains on examples already seen, skips unseen examples

Why this is harmful:

• Silently biases your data distribution
• Model still trains and completes
• Problem only visible when final metrics disappoint
• Gets worse with more restarts (which scale makes routine)

The Complete Checkpoint Must Include:

Checkpoint = {
  model weights,
  optimizer state,
  training step,
  data position (sample/shard offset),  ← often forgotten
  RNG seeds and states                   ← often forgotten
}

Why RNG state matters: Shuffling and augmentation use random number generators. Without saving their state, the data order after restart won't match the order before — a saved position points at the wrong samples.

---

Step 9: Putting It All Together

Here is the complete mental model:

Hardware failure occurs
        ↓
Auto-recovery finds last valid checkpoint (.metadata present)
        ↓
Restores: model + optimizer + step + data position + RNG state
        ↓
Training resumes with minimal lost work (small checkpoint interval)
        ↓
Data pipeline overlaps loading with compute (no GPU starvation)
        ↓
async_save checkpoints frequently with minimal GPU pause
        ↓
Repeat → High goodput maintained throughout

---

Summary: Decision Checklist

Unifying principle: Frequent + inexpensive + complete checkpoints turn hardware failure from a catastrophe into a minor inconvenience. An overlapped data pipeline keeps GPUs busy in between.