After studying this material, students should be able to:

1. Explain the limitations of Visual Chain-of-Thought (Visual CoT) reasoning in video understanding
2. Describe what Internalized Visual Thinking (IVT) is and how it works
3. Compare Visual CoT vs IVT across key dimensions (accuracy, efficiency, inference)
4. Understand the concept of "internalizing" world modeling during training
5. Evaluate trade-offs in multimodal reasoning system design

---

Step-by-Step Teaching

Step 1: Foundation — What is Chain-of-Thought (CoT) Reasoning?

The Basic Idea

Imagine you ask someone: "What will happen next in this video?"

A naive model just outputs an answer directly.

A CoT model thinks step-by-step first:

Step 1: I see a person reaching for a cup
Step 2: The cup is near the edge of the table
Step 3: Their hand trajectory suggests they'll knock it
Answer: The cup will fall

Key Insight: Breaking reasoning into intermediate steps improves accuracy — this is Chain-of-Thought.

---

Step 2: What is Visual CoT?

Extending CoT to Visual Reasoning

Visual CoT takes this further — instead of only generating text steps, it generates intermediate images as reasoning steps.

Input Video (partial) 
        ↓
[Generate Future Frame Image 1] → encode → reason
        ↓
[Generate Future Frame Image 2] → encode → reason
        ↓
Final Text Answer

Why Visual CoT Was Appealing

---

Step 3: The Problem with Visual CoT

The Core Bottleneck

Visual CoT introduces substantial inference overhead. Let's break down why:

For EVERY question at inference time:

1. Generate future frame image (expensive pixel synthesis)
2. Re-encode that image back into the model (expensive encoding)
3. Repeat for multiple frames
4. Finally produce answer

= Very slow end-to-end pipeline

Why This Matters Especially for Video

Video reasoning is time-sensitive and proactive — meaning:

• You need answers quickly
• Videos have many frames
• Generating pixel-level images for each reasoning step multiplies the cost dramatically

Analogy: Imagine having to draw a detailed picture every time you wanted to think about what might happen next. That's what Visual CoT forces the model to do at inference time.

---

Step 4: The Central Research Question

The paper asks a profound question:

"Can models learn to think visually during training, while reasoning directly at inference?"

This separates two phases:

---

Step 5: Introducing IVT — Internalized Visual Thinking

The Core Concept

IVT is a post-training framework — meaning it takes an already-trained model and further trains it with a specific objective.

What IVT Does During Training

Input: Partially observed video (first N frames)
              ↓
Model jointly predicts:
    ├── Latent representations of future frames
    │   (NOT actual pixel images — compressed representations)
    └── Target textual answer
              ↓
Model learns to capture:
    • Motion patterns
    • Object transitions  
    • Interactions between objects
    • Latent intent of agents

Key Terminology Explained

---

Step 6: How IVT Works — The Mechanism

Training Phase (Learning to Think Visually)

VIDEO: [Frame 1][Frame 2][Frame 3]...[Frame N] → [Future Frame N+1][N+2]...

During training:
┌─────────────────────────────────────────┐
│  Model sees: Frames 1 to N              │
│  Model must predict:                    │
│    • Latent encoding of Frame N+1       │  ← Visual thinking internalized
│    • Latent encoding of Frame N+2       │  ← into model weights
│    • Correct text answer                │  ← Final output goal
└─────────────────────────────────────────┘

The model is jointly optimized for both objectives simultaneously — this forces it to build internal representations that capture visual dynamics.

Inference Phase (Direct Reasoning)

VIDEO: [Frame 1][Frame 2][Frame 3]...[Frame N]
              ↓
         IVT Model
              ↓
         Text Answer ← Generated directly, no image synthesis needed

The Magic: The visual thinking is now baked into the model's weights from training. It doesn't need to explicitly generate images anymore — it already "knows" how to reason about future states internally.

---

Step 7: Visual CoT vs IVT — Side-by-Side Comparison

VISUAL CoT INFERENCE:
─────────────────────
Video Input
    │
    ▼
Generate Future Frame (pixel space) ──→ Expensive!
    │
    ▼
Encode Frame Back to Embeddings ──────→ Expensive!
    │
    ▼
Generate Next Future Frame ────────────→ Expensive!
    │
    ▼
Final Answer

Total: SLOW (5x+ slower)


IVT INFERENCE:
──────────────
Video Input
    │
    ▼
Final Answer

Total: FAST (same as standard model)

Performance Comparison

---

Step 8: What the Model Learns to Internalize

IVT trains the model to capture four key types of visual knowledge:

1. Motion Patterns

Frame 1: Ball at position A
Frame 2: Ball at position B
Model learns: Ball trajectory → predicts position C internally

2. Object Transitions

Frame 1: Cup on table (intact)
Frame 2: Cup falling
Model learns: State change dynamics

3. Interactions

Frame 1: Person approaching door
Frame 2: Hand on handle
Model learns: Action-object relationships

4. Latent Intent

Frame 1: Person looking at phone
Frame 2: Person walking toward exit
Model learns: Goal-directed behavior patterns

---

Step 9: The Broader Theoretical Insight

World Modeling Internalized

The paper introduces an important conceptual shift:

OLD VIEW:
"To reason about future visual states, 
 you must generate them explicitly"

NEW VIEW (IVT):
"Future visual states can be internalized 
 as implicit knowledge during training,
 eliminating the need for explicit generation at inference"

This connects to the concept of predictive world modeling — the idea that intelligent systems build internal models of how the world works, rather than simulating it explicitly every time.

Human Analogy: When you catch a ball, you don't consciously simulate its trajectory frame-by-frame. Your brain has internalized physics from experience and acts directly. IVT does something similar for video models.

---

Step 10: Experimental Validation

The paper validates IVT across multiple dimensions:

What They Tested (Controlled Studies)

Key Results

IVT vs Text-Only Post-Training:
✓ IVT wins across ALL 6 evaluation settings

IVT vs Visual CoT:
✓ Comparable or better accuracy
✓ 5x+ reduction in end-to-end latency
✓ No image generation needed at inference

---

Summary: The Complete Picture

PROBLEM:
Visual CoT is accurate but slow due to 
pixel-space generation at inference time

INSIGHT:
Visual thinking can be internalized during 
training rather than executed at inference

SOLUTION (IVT):
Post-training framework that jointly optimizes:
  • Next-embedding prediction (visual thinking)
  • Text answer prediction (task performance)

RESULT:
Models that reason as accurately as Visual CoT
but 5x+ faster at inference time

BROADER LESSON:
Explicit pixel generation at inference may be 
unnecessary — world modeling can live in weights

---

Key Takeaways to Remember

1. Visual CoT generates intermediate images to reason — powerful but slow
2. IVT internalizes visual reasoning into model weights during training
3. At inference, IVT answers directly — no image generation needed
4. IVT achieves 5x+ speedup with comparable or better accuracy
5. This suggests predictive world modeling can be implicit, not explicit
6. The key innovation is joint optimization of visual prediction + text prediction during post-training