After studying this material, you should be able to:

1. Explain what Masked Diffusion Models (MDMs) are and how they differ from autoregressive models
2. Understand what tri-modal generation means and why it matters
3. Describe key design choices: scaling laws, modality mixing, noise schedules, batch-size effects
4. Explain the SDE-based reparameterization and why it solves the batch-size tuning problem
5. Evaluate the significance of the results at 3B model scale

---

Step-by-Step Teaching

Step 1: Foundation — What Problem Are We Solving?

Traditional Language Models (Autoregressive)

Input:  "The cat sat on the..."
Output: "mat"  ← predicts ONE token at a time, left to right

Limitations:

• Sequential generation = slow
• Struggles to natively handle non-text modalities (images, audio)
• Each modality typically needs a separate model

💡 Key Question: Can we build ONE unified model that generates text, images, AND audio together?

---

Step 2: What Are Diffusion Models?

The Core Idea of Diffusion

Diffusion models work in two phases:

FORWARD PROCESS (Training):
Clean Data → Add Noise Gradually → Pure Noise
   🐱      →      🌫️🐱        →    🌫️🌫️🌫️

REVERSE PROCESS (Generation):
Pure Noise → Remove Noise Gradually → Clean Data
  🌫️🌫️🌫️  →      🌫️🐱        →      🐱

Continuous vs. Discrete Diffusion

💡 Language is discrete (words/tokens), so we need Discrete Diffusion

---

Step 3: Masked Diffusion Models (MDM) Specifically

What is "Masking" in this context?

Instead of adding Gaussian noise, MDMs use masking as their noise process:

FORWARD (Masking):
"The cat sat on the mat"
→ "The [MASK] sat on [MASK] mat"
→ "[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]"

REVERSE (Unmasking):
"[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]"
→ "The [MASK] sat on [MASK] mat"
→ "The cat sat on the mat"

Why MDMs Over Autoregressive?

---

Step 4: Uni → Bi → Tri-Modal (The Paper's Contribution)

Evolution of Multimodal Models

UNIMODAL:          Text only
                      📝

BIMODAL:           Text + Images
(Previous work)       📝 + 🖼️

TRI-MODAL:         Text + Images + Audio
(This paper) →        📝 + 🖼️ + 🔊

What "Pretrained from Scratch" Means

Previous approaches:

Take existing text model → Fine-tune for images → Fine-tune for audio
(suboptimal — modalities added as afterthoughts)

This paper's approach:

Train on ALL THREE modalities simultaneously from the beginning
(modalities inform each other during learning)

💡 Analogy: Learning Spanish by only translating from English vs. growing up speaking both languages simultaneously

---

Step 5: The Four Key Design Dimensions Analyzed

5.1 Multimodal Scaling Laws

What are scaling laws? Scaling laws describe how model performance improves as you increase:

• Model size (parameters)
• Data size (tokens)
• Compute (FLOPs)

Performance
    ↑
    |        ● 3B params
    |      ●
    |    ●
    |  ●
    |●
    +----------------→ Scale

Why it matters for multimodal:

Does adding more modalities help or hurt scaling efficiency? The paper systematically measures this.

---

5.2 Modality Mixing Ratios

The core question: How much of each modality should be in training data?

Option A: 80% Text, 10% Image-Text, 10% Audio-Text
Option B: 33% Text, 33% Image-Text, 33% Audio-Text  
Option C: 60% Text, 30% Image-Text, 10% Audio-Text

Why this matters:

• Too much text → model forgets how to generate images/audio
• Too much image data → text quality degrades
• Finding the optimal ratio is non-trivial

💡 Analogy: A student studying for 3 exams must allocate study time wisely — too much time on one subject hurts the others

---

5.3 Noise Schedules

What is a noise schedule?

It controls how fast masking happens during training:

LINEAR SCHEDULE:
Time:    0%    25%    50%    75%    100%
Masked:  0%    25%    50%    75%    100%

COSINE SCHEDULE:
Time:    0%    25%    50%    75%    100%
Masked:  0%     5%    50%    95%    100%
(spends more time at intermediate noise levels)

Why it matters: Different schedules teach the model different things:

• Early masking levels → learn fine details
• High masking levels → learn global structure

---

5.4 Batch-Size Effects (The Novel Contribution)

This is the most technically novel part of the paper.

The Problem First

In stochastic gradient descent (SGD):

SMALL BATCH:
- High gradient variance (noisy updates)
- More updates per epoch
- Better generalization sometimes
- Slower wall-clock time

LARGE BATCH:
- Low gradient variance (stable updates)  
- Fewer updates per epoch
- Needs more careful tuning
- Faster wall-clock time (GPU efficient)

The tension:

Physical Batch Size  ≠  Logical Batch Size
(what fits on GPU)      (what's optimal for learning)

Previous work required manually tuning which batch size gives best results — expensive!

---

Step 6: The SDE Reparameterization (Key Innovation)

What is an SDE?

A Stochastic Differential Equation describes how a system evolves with both deterministic and random components:

dx = f(x,t)dt + g(t)dW
      ↑              ↑
 Deterministic    Random/Stochastic
   drift          component

How the Paper Uses SDEs to Solve Batch-Size Problem

The insight: The masking process in MDMs can be described as an SDE. By reparameterizing through this SDE framework:

BEFORE (Old approach):
Physical Batch Size → affects gradient variance → need to tune separately
     ↕ (coupled)
Logical Batch Size  → affects learning quality

AFTER (SDE Reparameterization):
Physical Batch Size ←→ DECOUPLED ←→ Logical Batch Size
(choose based on GPU)              (choose based on optimization)

Practical Meaning

Engineer can now say:
"I have 8 A100 GPUs → Physical batch = 512 (for GPU efficiency)"

AND separately:
"Optimal learning needs logical batch = 2048"

The SDE reparameterization handles the translation automatically!

💡 Analogy: Like having a car with automatic transmission — you choose your desired speed (logical), the transmission handles gear selection (physical) automatically

---

Step 7: Results at Scale

What Was Built

Model Scale:    3 Billion Parameters
Training Data:  6.4 Trillion Tokens
Modalities:     Text + Image + Audio

Three Tasks Demonstrated

Why "Strong Results" Matters

Previous SOTA approach:    Separate specialized models per modality
                                📝model  🖼️model  🔊model

This paper:                ONE unified model
                                    🤖
                           achieving competitive results
                           across ALL THREE tasks

---

Step 8: Putting It All Together

The Complete Picture

                    TRI-MODAL MDM DESIGN SPACE
                    
    DATA LAYER:
    [Text Data] + [Image-Text Data] + [Audio-Text Data]
         ↓              ↓                    ↓
    Mixing Ratio: (optimized through systematic study)
    
    MODEL LAYER:
    Masked Diffusion Model (3B parameters)
    - Noise Schedule: (optimized)
    - Batch Size: (handled by SDE reparameterization)
    
    INFERENCE LAYER:
    - Optimized sampling defaults provided
    
    OUTPUT:
    Text Generation | T2I | T2S

---

Summary: Key Takeaways

---

Self-Check Questions

1. Why is masking a natural noise process for discrete (text) data?
2. What is the difference between physical and logical batch size?
3. Why is training from scratch on all modalities better than fine-tuning?
4. What does the SDE reparameterization eliminate the need for?
5. Name the three tasks the final model can perform.