After studying this material, you should be able to:
- Explain why GPU reliability is uniquely challenging for large-scale ML workloads
- Describe the four-layer reliability framework and what each layer addresses
- Distinguish between detection, remediation, and recovery strategies
- Understand how observability and attribution enable safe automation
- Evaluate tradeoffs between different remediation approaches
Step-by-Step Study Material
Step 1: Why GPU Reliability Is a Unique Problem
The Core Challenge
Before diving into solutions, you need to understand why this problem is hard.
Traditional Server Job:
[Task A] runs independently → fails → restart just Task A ✓
GPU Training Job (Gang Scheduling):
[GPU 1] ←→ [GPU 2] ←→ [GPU 3] ←→ ... ←→ [GPU 400]
All move in lockstep. One fails = EVERYTHING stops ✗
Key Properties That Make This Hard
| Property | Implication |
|---|---|
| Jobs run for weeks | More time = more exposure to hardware failure |
| Hundreds of GPUs per job | More components = higher probability something breaks |
| Gang scheduling | Slowest participant sets the pace for everyone |
| Hardware can silently degrade | A GPU can lie about its health |
| Standard checks are insufficient | "Instance is up" ≠ "GPU is healthy" |
What Standard Cloud Checks Miss
Cloud Provider Checks:
✓ Is the machine powered on?
✓ Is the OS reachable?
✓ Is storage attached?
What they MISS:
✗ Is each GPU's on-chip memory healthy?
✗ Are GPU interconnects working?
✗ Is the high-speed network fabric intact?
✗ Are there thermal issues slowing a GPU down?
Key Insight: A machine can pass every standard check and still silently destroy a multi-week training job.
Step 2: The Four-Layer Reliability Framework
Think of this as defense in depth — each layer catches what the layer below cannot.
┌─────────────────────────────────────────────┐
│ Layer 4: OBSERVABILITY │
│ "Are the other three layers working?" │
├─────────────────────────────────────────────┤
│ Layer 3: TRAINING LOOP │
│ "Survive hardware loss cheaply" │
├─────────────────────────────────────────────┤
│ Layer 2: SCHEDULER & PLATFORM │
│ "Validate before assigning work" │
├─────────────────────────────────────────────┤
│ Layer 1: NODE & HARDWARE HEALTH │
│ "Find bad hardware, remove it" │
└─────────────────────────────────────────────┘
Layer 1 — Node and Hardware Health
Goal: Detect bad GPUs, network adapters, interconnects, memory errors, and thermal issues.
How it works:
Every 30 seconds on every GPU machine:
┌──────────────────────────────────────────┐
│ Node-Problem-Detector Agent │
│ │
│ GPU Checks (via NVIDIA DCGM): │
│ • Driver-reported error codes │
│ • Uncorrectable memory errors │
│ • Interconnect integrity │
│ • Thermal state │
│ │
│ Network Checks: │
│ • Count of Elastic Fabric Adapters │
│ • Link state of each adapter │
└──────────────────────────────────────────┘
↓ Results become
┌──────────────────────────────────────────┐
│ Kubernetes Node Conditions │
│ (same mechanism as "NotReady" status) │
└──────────────────────────────────────────┘
Two-tier response system:
Check Fails
│
▼
PROBLEM CONDITION
• Blocks NEW work from landing
• Existing jobs continue running
• Wait for tolerance window
│
├── Clears on its own? → Resume normal operation
│
└── Persists beyond window?
▼
FAILURE CONDITION
• Evacuate the machine
• Drain all tasks
• Terminate instance
• Replace with new machine
Why not immediately terminate? Pulling a running job off a machine is disruptive. Many problems clear on their own. The tolerance window avoids unnecessary disruption.
Layer 2 — Scheduler and Platform
Goal: Validate machines before they receive work, and handle failures gracefully.
Machine lifecycle with health gates:
Machine Boots
│
▼
System checks pass? ──No──→ Do not release
│ Yes
▼
GPUs visible + topology correct? ──No──→ Do not release
│ Yes
▼
GPU diagnostics pass? ──No──→ Do not release
│ Yes
▼
Wait for health checks to run TWICE
(some checks like network adapter count
take more than one cycle to complete)
│
▼
Remove startup taint → Machine enters pool
│
▼
Job assigned → Pre-job validation:
• Short collective communication run
• Pairwise network tests
• Any failure here = job not admitted
Key Insight: A machine that fails within the first minute of a job fails before the job is admitted, not after wasting hours of compute.
On teardown:
Machine being retired
│
▼
Run deep diagnostics
│
▼
Record GPU hardware identifiers (immutable IDs
that persist across reboots and relaunches)
│
▼
Add to bad-hardware list
│
▼
Prevents same physical GPU from
quietly rejoining pool under new instance ID
Layer 3 — Training Loop
Goal: Make hardware loss cheap rather than catastrophic.
Two mechanisms:
Mechanism 1: Checkpointing
─────────────────────────
Week 1 Week 2 Week 3
│ │ │
▼ ▼ ▼
[ckpt] [ckpt] [ckpt] ← saved state
Machine fails mid-Week 3:
↓
Resume from Week 3 checkpoint
↓
Lose hours, not weeks ✓
Mechanism 2: Elastic Workers
────────────────────────────
Normal: 400 GPUs working
Machine fails: 396 GPUs continue
Scheduler retries failed tasks
Job continues (possibly slower) ✓
Layer 4 — Observability
Goal: Know whether the first three layers are actually working.
Three Service-Level Objectives (SLOs), each split by cause:
SLO Miss
│
├── Hardware cause? → Layer 1 problem
├── Driver cause? → Layer 1/2 problem
├── Platform cause? → Layer 2 problem
└── User cause? → Not a platform issue
Why split by cause? An availability dip from bad hardware requires a different response than one caused by a scheduler change or user misconfiguration.
Key metric — Goodput:
Uptime metric (misleading):
Job running = ✓ healthy
(Even if one slow GPU is bottlenecking 399 others)
Goodput metric (accurate):
Useful work completed / Total GPU hours consumed
(Captures the gang-scheduling bottleneck problem)
Step 3: Graded Remediation
Not every problem requires the same response. Matching action to fault minimizes disruption and restores capacity faster.
Fault Severity Action Time to Restore
──────────────────────────────────────────────────────
Software wedge → Kill job Minutes
Transient fault → Reset single GPU Minutes
Firmware drift → Reboot machine ~10-15 minutes
Hardware fail → Terminate + replace Hours
Why this matters: A full node replacement is the slowest path. Graded remediation means most faults are resolved in minutes, not hours.
The bad-hardware tracking system:
GPU fails hardware test
│
▼
Record immutable hardware ID
(persists across: reboots, relaunches, new instance IDs)
│
▼
Report to cloud provider
│
▼
Maintain internal bad-hardware list
│
▼
GPU cannot silently rejoin pool
under a different instance ID
Step 4: Attribution — Making Automation Safe
This is a subtle but critical concept. Why not just retry every failure?
Without attribution:
Failure → Retry → Failure → Retry → ...
↑
Burning GPU hours on
unfixable application bugs
Masking real issues
With attribution:
Failure → Classify cause
│
├── Platform fault? → Retry automatically ✓
└── Application fault? → Surface to user ✗ (don't retry)
Real example from the article:
Symptom: NCCL timeouts (collective communication library)
Initial interpretation: Network fault → cycle machines
Actual cause: Data-loading stalls at high worker counts
Without attribution:
→ Healthy machines cycled repeatedly
→ Problem never fixed
→ GPU hours wasted
With attribution:
→ Classified as application issue
→ Fix applied to data pipeline
→ Problem resolved ✓
Step 5: The Complete Reliability Loop
Now you can see how all pieces connect:
┌─────────────────────┐
│ Machine Allocated │
└──────────┬──────────┘
│
┌──────────▼──────────┐
│ Pre-job Validation │
│ • Collective comms │
│ • Network tests │
└──────────┬──────────┘
│ Pass
┌──────────▼──────────┐
│ Job Running │◄──────────────┐
│ Passive monitoring: │ │
│ • Driver errors │ │
│ • Memory errors │ │
│ • Thermal events │ │
│ • Slow rank detect │ │
└──────────┬──────────┘ │
│ Problem detected │
┌──────────▼──────────┐ │
│ Classify Fault │ │
└──────────┬──────────┘ │
│ │
┌────────────────┼────────────────┐ │
│ │ │ │
┌─────────▼──────┐ ┌───────▼──────┐ ┌──────▼──────┐ │
│ Kill job │ │ Reset GPU │ │ Replace node│ │
│ (software │ │ (transient │ │ (hardware │ │
│ wedge) │ │ fault) │ │ failure) │ │
└─────────┬──────┘ └───────┬──────┘ └──────┬──────┘ │
│ │ │ │
└────────────────┴───────────────┘ │
│ │
┌──────────▼──────────┐ │
│ Resume from │ │
│ checkpoint ├───────────────┘
└─────────────────────┘
Step 6: Advanced Concepts and Future Directions
GPU Partitioning for Inference (MIG)
Without MIG:
┌─────────────────────┐
│ Full GPU │ ← Small inference job wastes most of this
└─────────────────────┘
With MIG (Multi-Instance GPU):
┌──────┬──────┬───────┐
│ Job A│ Job B│ Job C │ ← Isolated partitions, efficient use
└──────┴──────┴───────┘
Fault in A does NOT affect B or C
Temperature as a Performance Fault
Standard thinking: GPU within thermal limits = healthy
Better thinking:
GPU running 10-15°C hotter than peers
│
▼
Sets pace for entire gang (remember: slowest participant)
│
▼
Treat large temperature gap as a fault
│
▼
Stop job, move to cooler GPUs
Kubernetes Dynamic Resource Allocation (DRA) — Future
Current limitation:
Bad GPU on a machine → Lose entire machine
With DRA:
Bad GPU on a machine → Isolate that one GPU
→ Keep remaining GPUs serving
→ Especially valuable for inference
(many small jobs share a machine)
Summary: Mental Model
Think of this system as a hospital triage system for GPUs:
PREVENTION (Layers 1 & 2):
"Don't let sick patients into the operating room"
→ Validate hardware before assigning work
→ Continuously monitor during work
→ Remove bad hardware automatically
RESILIENCE (Layer 3):
"If something goes wrong mid-surgery, have a recovery plan"
→ Checkpoints = surgical notes saved throughout
→ Elastic workers = backup surgeons ready
ACCOUNTABILITY (Layer 4):
"Track outcomes to improve the system"
→ SLOs split by cause
→ Attribution before retry
→ Goodput over uptime
Quick Reference: Key Terms
| Term | Definition |
|---|---|
| Gang scheduling | All workers in a job must run together; one failure stops all |
| Goodput | Useful work completed, not just "time running" |
| DCGM | NVIDIA's tool for GPU health monitoring |
| MIG | Multi-Instance GPU — partitioning one GPU for multiple tenants |
| NCCL | Library GPUs use to communicate during distributed training |
| Elastic Fabric Adapter (EFA) | High-speed network interface for cross-machine GPU communication |
| Karpenter | Kubernetes node lifecycle manager, used here for auto-repair |
| DRA | Dynamic Resource Allocation — future Kubernetes feature for device-level allocation |
| Checkpoint | Saved job state allowing resume after failure |
| Xid errors | NVIDIA GPU error codes indicating specific hardware faults |