Step-by-Step Teaching
Step 1: Understand the Core Problem
Why Memory Is Under Pressure
Modern AI systems face a growing memory crisis driven by three forces:
| Force | What It Means |
|---|---|
| Larger models | Trillions of parameters must be stored |
| Longer contexts | Agentic tasks process up to 3 million tokens per request |
| RAG applications | Enterprise data added to prompts increases working memory |
Two things compete for the same limited GPU memory (HBM):
- Model weights — the learned values that define what the AI knows
- KV cache — the working memory that holds context so the model doesn't reprocess the same information repeatedly
🔑 Key Insight: More demand + limited supply = memory pressure. This drives up cost and limits how many users can be served simultaneously.
Step 2: Understand What HBM Actually Is
The Warehouse Analogy
Think of GPU memory this way:
HBM = Warehouse
Memory Bandwidth = Loading Docks
- A bigger warehouse holds more inventory
- But loading docks control how fast things move in and out
- Once docks are at full capacity, adding warehouse space doesn't help throughput
- Extra space still costs money and draws power — wasted resources
This ceiling is called the memory wall — the point where bandwidth, not capacity, becomes the bottleneck.
🔑 Key Insight: Simply adding more HBM is not the solution. You hit a bandwidth ceiling before the extra capacity becomes useful.
Step 3: Understand the Memory Hierarchy
Not All Storage Is Equal
Because HBM has limits, AI factories use a tiered storage system:
FASTEST / MOST EXPENSIVE
↑
GPU HBM ← Model weights + active KV cache
CPU Memory ← Paused/inactive context
Local SSDs ← Overflow storage
Shared Storage ← Large-scale, low-cost capacity
↓
SLOWEST / CHEAPEST
The principle: Keep what's actively needed in the fastest tier. Move everything else to cheaper, slower tiers and bring it back before it's needed.
🔑 Key Insight: Efficient AI infrastructure is about placing data on the right tier at the right time — not just buying more of the fastest tier.
Step 4: Understand Full-Stack Codesign
What It Means
Instead of optimizing one component in isolation, full-stack codesign means designing these layers together so they work as a unified system:
GPU + CPU + Networking + Storage + Software
Each layer is designed to hand off work efficiently to the next, eliminating waste at every transition.
🔑 Key Insight: The goal is to maximize useful work per byte across the entire system, not just maximize specs on a single component.
Step 5: Learn the Three Optimization Techniques
Technique 1: Shrink and Distribute Model Weights
Problem: Model weights consume large amounts of HBM, leaving less room for KV cache.
Solutions:
| Method | How It Helps |
|---|---|
| NVFP4 precision format | Stores weights in 4-bit format instead of 8-bit (FP8), cutting weight memory roughly in half |
| Distributing across 72 GPUs (NVL72) | Each GPU holds only a fraction of the total model, freeing HBM for KV cache |
Result: More HBM available for serving concurrent users.
Technique 2: Shrink KV Cache and Offload Inactive Context
Problem: KV cache grows with longer conversations and agentic tasks, consuming HBM even when context is temporarily idle (e.g., during tool calls).
Solutions:
| Method | How It Helps |
|---|---|
| NVFP4 for KV cache | Reduces KV cache footprint so more context fits in HBM |
| NVIDIA Vera CPU | Executes tools faster (less idle time) + provides nearby LPDDR memory for paused context |
| NVIDIA CMX storage | Offloads inactive context to shared, low-cost storage optimized for short-lived data |
| Dynamo KV Cache Reuse | Manages KV cache as one unified resource across GPU, CPU, and storage — context computed once is reused system-wide |
Result: HBM is freed from holding idle context, increasing active user capacity.
Technique 3: Keep RAG Data Outside HBM
Problem: RAG systems build large searchable indexes of enterprise documents that would consume enormous HBM if stored there.
Solutions:
| Method | How It Helps |
|---|---|
| Nemotron Retriever models | Produce smaller embeddings with higher accuracy — less data to store |
| cuVS library | Builds and serves the search index on CPU memory and storage, never in HBM |
Result: Only the specific passages the model actually needs enter HBM — the full knowledge base never does.
Step 6: See the Results
Why This Matters in Practice
The metric AI factory operators care about most is:
Tokens per second per dollar (within a power budget, at acceptable response times)
| Comparison | HBM Capacity Change | Throughput per Dollar Improvement |
|---|---|---|
| H200 → GB300 NVL72 | ~2x more HBM | 10x better |
| GB300 → Vera Rubin NVL72 | ~3% more HBM | 30x better |
🔑 Critical Takeaway: A 3% increase in HBM capacity produced a 30x improvement in throughput per dollar. This is only possible through full-stack codesign — not by simply adding memory.
Summary: The Complete Mental Model
PROBLEM: HBM is fast but limited → model weights + KV cache compete for space
↓
WRONG FIX: Just add more HBM → hits bandwidth wall, wastes cost/power
↓
RIGHT FIX: Full-Stack Codesign across the memory hierarchy
↓
┌─────────────────────────────────────────┐
│ Shrink weights → NVFP4 + NVL72 │
│ Shrink KV cache → NVFP4 + offload │
│ Keep RAG off HBM → cuVS + Nemotron │
└─────────────────────────────────────────┘
↓
RESULT: More useful tokens per dollar per watt
Every optimization targets a different source of HBM waste, and together they compound into dramatic efficiency gains across generations.