After studying this material, you should be able to:

1. Explain what Mixture-of-Experts (MoE) models are and why they matter
2. Describe the core computational challenges of training large MoE models
3. Identify the key parallelism techniques used to distribute MoE training across hardware
4. Understand the optimization strategies that improve training efficiency
5. Interpret benchmark results and performance trade-offs in large-scale model training

---

Step-by-Step Teaching

Step 1: The Problem — Why Training Large AI Models Is Hard

Before understanding the solution, you need to understand the problem.

Training large language models requires enormous compute resources, which means:

• Very high financial costs
• High energy consumption
• Only large organizations can afford it

Key insight: The bigger the model, the more parameters it has, and the more computation every single input requires.

---

Step 2: What Is a Mixture-of-Experts (MoE) Model?

A standard dense model activates all parameters for every input token.

A Mixture-of-Experts (MoE) model works differently:

Input Token → Router → Selects a few "Experts" → Only those experts compute

Key Terminology

Concrete Example from the Article

Pool grew from 8 → 128 experts Still only 4 experts selected per token Active parameters stayed fixed at ~3.2 billion Total capacity grew from 4.6B → 47B parameters Training throughput dropped by less than 5%

Why this matters: You get a much larger, more capable model without proportionally increasing computation per token.

---

Step 3: The Hidden Costs of MoE — Why It's Not Free

MoE sounds perfect, but it introduces new challenges:

Challenge 1: Memory Storage

Even though only a few experts are used per token, the entire model must still be stored across GPU memory.

Challenge 2: Communication Overhead

Routing tokens to the right experts across a GPU cluster creates coordination costs:

GPU 1 has Expert A
GPU 7 has Expert B
Token needs both → Data must travel between GPUs → Latency

Challenge 3: Routing Efficiency

Deciding which expert handles which token must happen fast and accurately, or the efficiency gains disappear.

Core tension: As MoEs grow larger, communication and coordination costs can erase the computational advantage of sparse activation.

---

Step 4: How OLMo-Core 3 Solves These Problems — Parallelism Techniques

OLMo-Core 3 uses three distribution strategies to split the model across hardware:

4a. Expert Parallelism

GPU 1: Experts 1–32
GPU 2: Experts 33–64
GPU 3: Experts 65–96
GPU 4: Experts 97–128

Each GPU stores only a portion of the expert pool, not all of them.

4b. Pipeline Parallelism

A language model has many layers (successive transformation stages).

Layer 1–10  → GPU Group A
Layer 11–20 → GPU Group B
Layer 21–30 → GPU Group C

Each GPU group handles a slice of the model's depth, reducing per-GPU memory requirements.

4c. Distributed Optimizer

During training, the optimizer stores extra data to calculate weight updates (momentum, variance, etc.).

• Old approach: Every GPU stores a full copy of optimizer state
• New approach: Optimizer state is spread across GPUs

Combined effect: No single GPU needs to hold the entire model, its layers, AND its training state simultaneously.

---

Step 5: Optimization Techniques — Making Routing and Computation Faster

Beyond distribution, OLMo-Core 3 adds three efficiency optimizations:

5a. Rowwise Expert Parallelism

Routed data is placed directly into expert input buffers, minimizing data rearrangement overhead.

Without: Token → Rearrange → Buffer → Expert
With:    Token → Buffer → Expert  ✓ (fewer steps)

5b. GPU-Resident Routing

Routing metadata (which expert handles which token) stays on the GPU.

• Old problem: CPU had to wait for routing info to be copied back before queuing work
• New approach: CPU can queue work immediately → less idle time

5c. Grouped GEMM

GEMM = General Matrix Multiplication (the core math operation in neural networks)

Each expert handles a small batch → many small matrix operations.

Without: Small op → Small op → Small op → Small op (inefficient)
With:    [Group all small ops] → One efficient large operation ✓

---

Step 6: MXFP8 — Precision Trade-offs

Numbers in computers can be stored with varying levels of precision:

What the Article Found

Using MXFP8 where it helped most:

• Training throughput +21% faster than BF16
• Peak active memory dropped from 103 GiB → 95 GiB
• Most gains came from feed-forward computation and data movement between experts

The Trade-off to Understand

Fewer bits → Less data to move ✓
BUT
Fewer bits → Must convert formats → Conversion costs time ✗

Key principle: MXFP8 only helps when the savings from moving less data outweigh the cost of converting between formats.

---

Step 7: Switching from FSDP to DDP — A Critical Architecture Decision

This is an important systems-level change worth understanding:

Old Approach: FSDP (Fully Sharded Data Parallelism)

For each mini-batch:
  Gather weights from all GPUs → Compute → Reshard weights → Repeat

Problem: Constant weight gathering = expensive repeated communication

New Approach: DDP (Distributed Data Parallelism)

Experts stay resident on their assigned GPUs
Data is routed TO the experts (not the other way around)

Performance Result

On 8 NVIDIA B300 GPUs with a 47B parameter MoE:

---

Step 8: Surprising Findings — What Didn't Work as Expected

The article documents important negative results — things that seemed logical but failed:

Finding 1: Token Gerrymandering

A routing balance score improved while actual workload distribution got worse.

Lesson: Metrics can be gamed or misleading. Always verify what you're actually measuring.

Finding 2: Lower Learning Rates for Experts Didn't Help

Logic: Experts see fewer tokens → reduce their learning rate to compensate Reality: This did not improve results in their tests.

Lesson: Intuitive adjustments don't always transfer across model families.

Finding 3: Input Values Affect GPU Timing

Same matrix dimensions, different input values → different computation times.

Lesson: Benchmarks must match both shape AND input values to be valid comparisons.

Finding 4: More Overlap ≠ More Speed

Overlapping communication and computation (running them simultaneously on separate GPU streams) sometimes slowed things down.

Lesson: System interactions are complex. More parallelism at the micro level can create bottlenecks elsewhere.

---

Step 9: Scale Achieved — Putting It All Together

---

Summary: The Complete Mental Model

PROBLEM: Large models are expensive to train
    ↓
SOLUTION: MoE — only activate part of the model per token
    ↓
NEW PROBLEM: Routing and communication create overhead at scale
    ↓
OLMO-CORE 3 RESPONSE:
  ├── Expert Parallelism     → Distribute expert storage
  ├── Pipeline Parallelism   → Distribute layer storage  
  ├── Distributed Optimizer  → Distribute training state
  ├── Rowwise EP             → Minimize data rearrangement
  ├── GPU-Resident Routing   → Eliminate CPU wait time
  ├── Grouped GEMM           → Batch small computations
  └── MXFP8                  → Reduce data movement cost
    ↓
RESULT: Trillion-parameter MoE training with manageable efficiency loss

---

Self-Check Questions

1. Why does an MoE model have more total parameters than active parameters?
2. What is the difference between expert parallelism and pipeline parallelism?
3. Why did switching from FSDP to DDP improve throughput?
4. Under what condition does MXFP8 actually help performance?
5. What is token gerrymandering and why is it a problem?