Concept 1: Why GPU Failures Are Inevitable at Scale
The Core Idea
At large scale, GPU failures aren't rare exceptions — they're mathematically guaranteed.
The Math
Using a conservative 1% annualized failure rate per GPU, the probability of at least one failure in a job is:
P(at least one failure) = 1 - (1 - failure_rate)^(N × T/365)
Where:
- N = number of GPUs
- T = job duration in days
What This Means in Practice
| Job Size | Duration | Failure Probability |
|---|---|---|
| 256 GPUs | 30 days | ~19% |
| 1,024 GPUs | 30 days | ~57% |
Key Takeaway
The question isn't if a failure will happen — it's when, and whether your system can handle it.
Concept 2: The Three Categories of GPU Failures
Why This Matters
Not all failures are equal. They differ in how visible they are and how much damage they cause before detection.
Category 1: Crashed Jobs ✅ (Visible but Uninformative)
What happens:
- A GPU falls off the bus, fabric fails, or a rank diverges
- Every GPU blocks on the same collective operation
- A watchdog timer eventually kills the entire job
The problem:
Error: NCCL watchdog timeout
This message tells you the symptom, not the cause. The actual root cause could be:
- Hardware degradation
- Network fabric issues
- Filesystem hang
- Software bug
Recovery: Restart from last checkpoint
Category 2: Silent Slowdowns ⚠️ (Invisible Performance Drain)
What happens:
- A degraded GPU keeps training, logs look normal, loss still decreases
- BUT: the entire job runs at the speed of the slowest GPU
Causes:
| Signal | Cause |
|---|---|
HW_SLOWDOWN | General hardware throttling |
HW_THERMAL_SLOWDOWN | Overheating |
| Link downgrade | Persistent interconnect errors |
| Memory bandwidth drop | Accumulated memory faults |
Why it's dangerous: Wastes compute and money for hours without any obvious alert
Category 3: Numerical Corruption 🔴 (Silent and Destructive)
What happens:
- Memory faults that ECC (Error Correction Code) cannot fix propagate through training
- The model trains on corrupted values
How it surfaces:
- NaN (Not a Number) loss values
- Unstable convergence
- Model quality regressions discovered after deployment
Key distinction: ECC fixes many transient faults automatically, but not all
Concept 3: A Real-World Failure Case Study
The Incident
A training run crashed 7 hours in with a NCCL timeout.
Root Cause Investigation
The culprit was a single InfiniBand port that went down once and recovered — never flapping again.
The Two-Timeout Problem
Stack Layer | Timeout | Default Duration
---------------------|----------------------|------------------
PyTorch (top) | NCCL watchdog | ~10 minutes
InfiniBand (bottom) | NCCL_IB_TIMEOUT | ~7 seconds
The critical insight:
NCCL_IB_TIMEOUTfires long before the PyTorch watchdog. If a port stays down longer than ~7 seconds, the connection is already dead before the watchdog even notices.
The Lesson
- Wrong metric: Counting flap frequency
- Right metric: Measuring cumulative downtime per flap
A single long flap = same damage as many short flaps
The Fix
- Tuned
NCCL_IB_TIMEOUTdefaults for better resilience - Used port-down signals to proactively restart from checkpoint instead of waiting for the watchdog
Concept 4: The Three-Layer Health Check System (gpu-monitor)
The Core Design Principle
Different failures are catchable at different times, so checks run at different stages.
┌─────────────────────────────────────────────────────┐
│ Layer 1: Active Bootstrap Checks (node startup) │
├─────────────────────────────────────────────────────┤
│ Layer 2: Passive Continuous Checks (during jobs) │
├─────────────────────────────────────────────────────┤
│ Layer 3: Periodic Multi-Node Checks (between jobs) │
└─────────────────────────────────────────────────────┘
Layer 1: Active Bootstrap Checks
When: At node provisioning AND between every customer workload
Purpose: Catch deterministic failures — things a targeted test can reliably surface
What's checked:
- GPU hardware diagnostics
- Memory integrity
- Single-node interconnect health
- Storage I/O
Outcome if failed:
Node fails check → Quarantined → Reset → Re-tested → Return to fleet OR permanent removal
Every workload is guaranteed to start on a node that just passed the full suite
Layer 2: Passive Continuous Checks
When: Constantly, while workloads are running
Purpose: Catch non-deterministic failures that only emerge under sustained load
What's monitored:
- Thermal throttling signals (
HW_SLOWDOWN,HW_THERMAL_SLOWDOWN,HW_POWER_BRAKE) - InfiniBand port flapping
- ECC error accumulation
- Memory bandwidth degradation
Outcome if triggered:
Node flagged → Cordoned → Drained → Same quarantine process as Layer 1
Layer 3: Periodic Multi-Node Active Checks
When: Periodically, on idle nodes between customer workloads
Purpose: Validate inter-node fabric — issues no single node can detect alone
Why separate from Layer 1:
- More expensive tests that don't fit in provisioning time
- Can be preempted when customer workloads need the nodes
Concept 5: How NCCL Bandwidth Tests Work Across Payload Sizes
Why Payload Size Matters
NCCL uses different code paths for different message sizes, and hardware issues often appear in only one path.
The Three Regimes
Message Size | Protocol Used | Key Metric | Why
----------------|----------------------|-----------------|---------------------------
Small (KB) | LL / LL128 | p95 latency | Latency-dominated
Medium (MB) | Tree → Ring switch | p95 latency | Algorithm transition point
Large (MB-GB) | Chunking/pipelining | BusBW | Bandwidth-dominated
Understanding the Two Bandwidth Metrics
| Metric | What It Measures |
|---|---|
| AlgBW (Algorithm Bandwidth) | Throughput as the application sees it |
| BusBW (Bus Bandwidth) | Actual link utilization (accounts for data moving multiple times across fabric) |
BusBW is the better indicator of hardware health because all-reduce moves each byte across the fabric multiple times
Sample Pass/Fail Criteria
| Payload | Pass Criterion | What It Catches |
|---|---|---|
| 1 KB | p95 latency ≤ 250 µs | Latency spikes in low-latency path |
| 16 MB | BusBW ≥ 50 GB/s AND p95 ≤ 750 µs | Algorithm transition issues |
| 1 GB | BusBW ≥ 250 GB/s | Bandwidth degradation at scale |
| 2 GB | BusBW ≥ 350 GB/s | Full pipelining health |
Summary: How the Concepts Connect
Scale makes failures inevitable (Concept 1)
↓
Failures come in 3 forms with different visibility (Concept 2)
↓
Real failures are complex and multi-layered (Concept 3)
↓
A 3-layer system catches failures at the right time (Concept 4)
↓
Fabric validation requires testing across payload sizes (Concept 5)
The Core Engineering Philosophy
Detect early, contain fast, recover cleanly — because at GPU scale, the system must be designed around the assumption that something is always failing somewhere.