MolmoMotion: Language-guided 3D motion forecasting | Ai2

Concept 1: The Core Problem — Perception vs. Prediction

What's the difference?

Why does prediction matter?

The article gives two concrete examples:

• A robot reaching for a cup must anticipate the cup's movement before touching it
• A video generator must know what realistic motion comes next to produce physically believable frames

Key insight: Predicting motion is harder than observing it, but far more useful in many real-world applications.

---

Concept 2: What MolmoMotion Actually Does

The basic input/output

INPUTS:
├── A video frame (what the scene looks like right now)
├── 3D points marked on an object (e.g., 8 dots on a bowl)
└── A written instruction (e.g., "Move and rotate the wooden bowl")

OUTPUT:
└── Predicted future 3D positions of those points over the next few seconds

A simple analogy

Think of placing tracking dots on a coffee cup, then asking the system: "If I slide this cup to the right, where will each dot end up?"

MolmoMotion answers that question — in 3D space — before the cup moves at all.

---

Concept 3: The Motion Representation — Why 3D Points?

The team needed a way to represent motion that works universally. They evaluated several options and chose object-attached 3D points in world space.

Why this representation specifically?

The article lists three required properties:

What this rules out

• Full video rendering — too expensive
• Human body templates — only works for humans
• 2D pixel tracking — changes when the camera moves

Key insight: Sparse 3D surface points are the only representation that satisfies all three requirements simultaneously.

---

Concept 4: The Model Architecture — How MolmoMotion Thinks

The backbone: Molmo 2

MolmoMotion is built on top of Molmo 2, a Vision-Language Model (VLM). This is important because it means the model already understands:

• What objects look like in images
• What language instructions mean
• How to connect words to visual regions

The processing pipeline

Step 1: Read the instruction → identify WHICH object is being referred to
Step 2: Locate the query points → find WHERE on the object they are
Step 3: Understand the motion → determine WHAT movement is described
Step 4: Predict trajectories → output WHERE each point will be over time

---

Concept 5: Two Variants — AR vs. FM

MolmoMotion comes in two flavors, each suited to different situations:

Variant 1: Autoregressive (MolmoMotion-AR)

Predicts coordinates one step at a time:
Time 0 → Time 1 → Time 2 → Time 3 → ...

• Treats 3D coordinates like text tokens — writes them out sequentially
• Each new position is conditioned on all previous positions
• Best for: Well-defined, predictable motions (e.g., sliding a bowl in a straight line)
• Strength: Smooth, accurate rollouts

Variant 2: Flow Matching (MolmoMotion-FM)

Starts with random noise → gradually transforms it into a trajectory

• Works in continuous 3D space
• Can represent multiple plausible futures simultaneously
• Best for: Ambiguous instructions where several outcomes are reasonable
• Strength: Handles uncertainty better

When to use which?

---

Concept 6: The Training Data Problem — MolmoMotion-1M

The challenge

To train a motion forecaster, you need:

• Large-scale videos ✓ (internet has millions)
• 3D point trajectories ✗ (didn't exist at scale)
• Action descriptions paired with specific objects ✗ (didn't exist)

The gap: Internet videos have scale and diversity, but no 3D annotations.

The solution: An automatic annotation pipeline

The team built a pipeline that automatically extracts 3D trajectories from regular videos:

Input: Video + action description
         ↓
Step 1: Ground the moving object (find it in the frame)
         ↓
Step 2: Sample query points on the object's surface
         ↓
Step 3: Track those points in 2D across frames
         ↓
Step 4: Lift 2D tracks into 3D metric space
         ↓
Step 5: Filter noisy/unreliable trajectories
         ↓
Step 6: Clip video to the window where the object actually moves
         ↓
Output: Clean 3D trajectories grounded to a specific object

Why filtering matters

Raw video tracks are noisy — depth estimation errors and tracking drift cause points to jitter and wander. The pipeline removes points that don't move consistently with the rest of the object.

The result: MolmoMotion-1M

• 1.16 million videos
• 736 motion types
• 5,600+ distinct objects
• Largest dataset of its kind ever assembled

---

Concept 7: The Evaluation Benchmark — PointMotionBench

Why a new benchmark was needed

Existing benchmarks measured whether generated motion looked plausible — a subjective standard. The team wanted to measure whether predicted trajectories actually match real future motion — an objective standard.

What PointMotionBench contains

Scene types covered

• Indoor manipulation (e.g., moving objects on a table)
• Egocentric hand-object interaction (e.g., first-person view of picking things up)
• Outdoor dynamic scenes (e.g., cars turning, animals walking)

How evaluation works

Given: Current frame + query points + action description
Predict: Future 3D trajectory
Compare: Predicted trajectory vs. actual trajectory
Score: Quantitative accuracy metrics

---

Concept 8: Downstream Application 1 — Robotics

The key insight

A human lifting a cup and a robot lifting a cup are very different actions, but the cup follows a similar path through 3D space in both cases.

This means motion knowledge learned from human videos can transfer to robot planning.

How it works

MolmoMotion predicts: "The cup will move up and to the right along this path"
Robot policy uses: That predicted path as a planning target

Results after fine-tuning on DROID (robot manipulation dataset)

Key insight: MolmoMotion doesn't just perform better — it learns faster, requiring far less robot training data.

---

Concept 9: Downstream Application 2 — Video Generation

The problem with standard video generation

When you give a text-to-video model an instruction like "A flamingo dips its beak into the water," the model has to guess what motion that implies. Vague instructions lead to vague or incorrect motion.

MolmoMotion's role

Standard pipeline:
Text instruction → Video generator → Video (motion may be wrong)

MolmoMotion pipeline:
Text instruction → MolmoMotion → Predicted 3D trajectories
                                        ↓
                              Trajectory-conditioned video generator → Video (motion follows instruction)

Results

MolmoMotion-guided video generation:

• Improves motion quality on all 5 motion-related metrics vs. the base model
• Beats a much larger image-to-video model on 4 out of 5 metrics

Key insight: Explicit 3D motion prediction acts as a bridge between language instructions and physically accurate video.

---

Concept 10: Limitations and Open Questions

Current limitations

The bigger vision

The article frames motion forecasting as fundamental to machine intelligence — as important as perception itself:

"We think forecasting — anticipating how objects in the world will move before they move — is as fundamental to machine intelligence as perceiving what's already there."

---

Summary: The Full Picture

PROBLEM: Machines can perceive motion but struggle to predict it

SOLUTION: MolmoMotion
├── Representation: Sparse 3D points in world space
├── Backbone: Molmo 2 (connects language to visual objects)
├── Two variants: AR (accuracy) and FM (uncertainty)
├── Training data: MolmoMotion-1M (1.16M videos, auto-annotated)
└── Benchmark: PointMotionBench (2.7K human-validated clips)

APPLICATIONS:
├── Robotics: 76.3% task success, 6x faster learning
└── Video generation: Better motion accuracy on 5/5 metrics

OPEN RELEASES:
├── Model weights
├── MolmoMotion-1M dataset
└── PointMotionBench benchmark

Each concept builds on the previous one: the problem motivates the representation, which shapes the architecture, which requires the data, which is measured by the benchmark, which validates the applications.