Diffusion vs. Autoregressive LMs: Speed and Scaling
Based on an article by Apple ML at the original source
Concept 1: Autoregressive Language Models (ARMs) — The Sequential Approach
What is an ARM?
An Autoregressive Language Model generates text one token at a time, where each new token depends on all previously generated tokens.
Input: "The cat sat on the"
Step 1: → "mat" (depends on all 5 input tokens)
Step 2: → "." (depends on all 6 tokens)
Step 3: → [END] (depends on all 7 tokens)
The Core Mechanism
P(token₅ | token₁, token₂, token₃, token₄)
↑
Conditioned on ALL previous tokens
The Key Problem: Sequential Dependency
Because each token must wait for the previous one:
Token 1 → Token 2 → Token 3 → Token 4 → Token 5
↑ ↑ ↑ ↑
Must Must Must Must
finish finish finish finish
first first first first
⚠️ This creates a bottleneck: You cannot parallelize generation across token positions.
What is Arithmetic Intensity?
Arithmetic Intensity = Compute Operations ÷ Memory Accesses
High Arithmetic Intensity = doing lots of math per memory read
= GPU is kept BUSY
Low Arithmetic Intensity = constantly waiting for memory
= GPU is IDLE most of the time
ARMs have LOW arithmetic intensity because:
- Each step generates only one token
- The GPU does very little math per memory access
- Most time is spent waiting for memory, not computing
Concept 2: Diffusion Language Models (DLMs) — The Parallel Approach
What is a DLM?
A Diffusion Language Model generates all tokens simultaneously through an iterative denoising process.
How Diffusion Works (Step by Step)
Phase 1 — Forward Process (Training):
Original Text: "The cat sat on the mat"
↓ Add noise
Step 1: "The [MASK] sat on [MASK] mat"
↓ Add more noise
Step 2: "[MASK] [MASK] sat [MASK] [MASK] mat"
↓ Add more noise
Step 3: "[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]"
Phase 2 — Reverse Process (Generation/Inference):
Start: "[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]"
↓ Denoise ALL positions at once
Step 1: "The [MASK] [MASK] on [MASK] mat"
↓ Denoise ALL positions at once
Step 2: "The cat sat on the mat"
✅ Key advantage: All token positions are processed in parallel at each step.
Why DLMs Have Higher Arithmetic Intensity
ARM per step: Process 1 token position
→ Small matrix operations
→ Low compute per memory access
DLM per step: Process ALL token positions simultaneously
→ Large matrix operations
→ High compute per memory access
Arithmetic Intensity Comparison:
ARM: ████░░░░░░░░░░░░ (low — memory bound)
DLM: ████████████░░░░ (higher — more compute per access)
Concept 3: The Scaling Problem — Why DLMs Struggle with Long Contexts
The Context Length Problem
Even though DLMs process tokens in parallel, they have a hidden scaling problem.
Understanding the Trade-off
Short Sequence (e.g., 128 tokens):
DLM processes all 128 tokens at once → HIGH arithmetic intensity ✅
Long Sequence (e.g., 4096 tokens):
DLM processes all 4096 tokens at once →
- Massive memory requirement 📈
- Attention cost grows QUADRATICALLY with length
- Arithmetic intensity DROPS ❌
Why Attention is the Culprit
The attention mechanism (used in both ARMs and DLMs) has this cost:
Attention Cost ∝ Sequence_Length²
128 tokens → 128² = 16,384 operations
512 tokens → 512² = 262,144 operations
4096 tokens → 4096² = 16,777,216 operations
As sequence grows:
Memory cost ↑↑↑
Compute cost ↑↑
─────────────────────────
Arithmetic Intensity ↓↓
⚠️ The paradox: DLMs gain intensity from parallelism across tokens, but lose it because longer sequences make attention increasingly memory-hungry.
Concept 4: Block-Wise Decoding — The Solution for DLMs
The Core Idea
Instead of generating all tokens at once, generate them in fixed-size blocks:
Full Sequence Diffusion (problematic):
[Token 1 ... Token 4096] ← all at once, huge memory cost
Block-Wise Diffusion (solution):
Block 1: [Token 1 ... Token 256] ← diffuse this block
Block 2: [Token 257 ... Token 512] ← diffuse this block
Block 3: [Token 513 ... Token 768] ← diffuse this block
...and so on
Why This Helps
┌─────────────────────────────────────────────┐
│ Block-Wise Decoding Benefits │
├─────────────────────────────────────────────┤
│ ✅ Block size is FIXED (e.g., 256 tokens) │
│ ✅ Arithmetic intensity stays CONSTANT │
│ ✅ Memory cost stays MANAGEABLE │
│ ✅ Scales to long contexts like ARMs do │
└─────────────────────────────────────────────┘
Decoupling Intensity from Sequence Length
Without block-wise:
Arithmetic Intensity = f(total sequence length) ← BAD, degrades
With block-wise:
Arithmetic Intensity = f(block size) ← GOOD, stays stable
💡 Key insight: Block-wise decoding makes DLM performance predictable and scalable, regardless of total output length.
Concept 5: Batched Inference — Why ARMs Win on Throughput
What is Batched Inference?
Instead of processing one request at a time, process multiple requests simultaneously:
Single inference: [Request 1] → 1 output
Batched inference: [Request 1]
[Request 2]
[Request 3] → 3 outputs simultaneously
[Request 4]
Why ARMs Benefit More from Batching
For ARMs:
Batch of 4 sequences:
Step 1: Generate token 1 for ALL 4 sequences simultaneously
Step 2: Generate token 2 for ALL 4 sequences simultaneously
...
→ Parallelism across BATCH dimension is FREE
→ Arithmetic intensity scales with batch size ✅
For DLMs:
DLM already uses parallelism across TOKEN positions
Adding batch parallelism creates COMPETITION for resources:
GPU Resources: [████████████████████████]
↑ ↑
Token parallelism Batch parallelism
(already using (now competing
most resources) for the rest)
Throughput Comparison
Throughput = Tokens Generated per Second
As Batch Size Increases:
ARM: ──────────────────────────────────────────►
Throughput scales well with batch size ✅
DLM: ──────────────────────────────────────────►
Throughput gains diminish faster ❌
(already parallelized across tokens)
🏆 ARMs win on throughput because they have more "room" to benefit from batch parallelism — their sequential token generation means the batch dimension is their primary source of parallelism.
Concept 6: Reducing Sampling Steps — The Key to DLM Latency
The Latency Problem
Latency = Total time to generate a complete response
ARM Latency:
Total = (Time per token) × (Number of tokens)
= small_value × N
DLM Latency:
Total = (Time per diffusion step) × (Number of diffusion steps)
= larger_value × S
Why Sampling Steps Matter So Much
DLM with 1000 steps:
[Denoise]→[Denoise]→[Denoise]→...×1000...→[Denoise] = SLOW ❌
DLM with 10 steps:
[Denoise]→[Denoise]→...×10...→[Denoise] = FAST ✅
The Quality vs. Speed Trade-off
More Steps: Higher quality output ←→ Higher latency
Fewer Steps: Lower latency ←→ Potentially lower quality
GOAL: Find minimum steps for acceptable quality
Comparing Latency
┌──────────────────────────────────────────────────┐
│ Latency Comparison │
├──────────────────────────────────────────────────┤
│ │
│ ARM: ████████████████ (scales with tokens) │
│ │
│ DLM ████████████████████████████ │
│ (many steps): (slow due to steps) ❌ │
│ │
│ DLM ████████ (competitive!) ✅ │
│ (few steps): │
│ │
└──────────────────────────────────────────────────┘
🔑 The key finding: Reducing sampling steps is the single most important optimization for DLMs to compete with ARMs on latency.
Summary: The Complete Picture
┌─────────────────┬──────────────────────┬──────────────────────┐
│ Dimension │ ARM │ DLM │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Generation │ Sequential │ Parallel │
│ Method │ (one token at a time)│ (all tokens at once) │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Arithmetic │ Low │ Higher │
│ Intensity │ (memory bound) │ (compute bound) │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Long Context │ Scales well │ Struggles │
│ Scaling │ ✅ │ ❌ (without blocks) │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Block-Wise Fix │ N/A │ Fixes scaling ✅ │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Batch │ Excellent │ Diminishing returns │
│ Throughput │ ✅ │ ❌ │
├─────────────────┼──────────────────────┼──────────────────────┤
│ Latency │ Competitive │ Needs fewer steps │
│ │ │ to compete │
└─────────────────┴──────────────────────┴──────────────────────┘
The Bottom Line
ARMs are battle-tested, efficient at scale, and win on throughput. DLMs offer a fundamentally different approach with higher arithmetic intensity, but need block-wise decoding + fewer sampling steps to truly compete in real-world deployment.