ToyBlocks: A Testbed for Robot Learning and Reasoning

Learning Outcomes

After studying this material, you should be able to:

1. Explain what FoundationWorlds are and why controlled simulation environments matter for robotics research
2. Describe how ToyBlocks is structured and what problems it solves
3. Distinguish between high-level and low-level models in robot learning
4. Understand how training data generation, evaluation, and sim-to-real transfer work
5. Analyze how agentic AI systems reason and act within robotic environments

---

Step 1: The Core Problem — Why Robotics Needs Controlled Environments

The Analogy First

Think of how biologists study complex systems:

Complex System → Controlled Environment → Vary conditions → Observe effects
Living organism → Petri dish          → Change variables → Draw conclusions

Robotics has the same need. Robots can fold laundry or sort parts, but:

• Demonstrations are narrow (work in one specific setting)
• It is unclear what the robot actually learned
• It is hard to trace why failures happen

What Researchers Actually Need

Key Insight: Without a controlled environment, robotics research is like doing chemistry without a lab — you can observe things, but you cannot isolate causes.

---

Step 2: Introducing FoundationWorlds

Definition

A FoundationWorld is a framework pairing:

Controlled Simulated World
        +
Automatic Task & Demo Generation Tools
        +
Metrics for Measuring Robot Behavior Over Time

What Questions FoundationWorlds Help Answer

• How do models improve as more training data becomes available?
• How do you best combine high-level reasoning with precise physical control?
• Does a model generalize to tasks it has never seen before?

---

Step 3: ToyBlocks — The First FoundationWorld

What It Is

ToyBlocks is a simulation built around:

• A robot arm
• Blocks on a tabletop
• Letters and pictures on block faces

The setting is deliberately simple, but it still requires the robot to:

• Reason about how blocks interact
• Push, flip, and stack blocks
• Search for hidden information (letters on unseen faces)

The Four Design Requirements

Why narrow scope? Open-ended domains make it impossible to fulfill all four requirements. ToyBlocks' narrow focus makes it possible to fulfill all of them completely.

---

Step 4: How ToyBlocks Generates Training Data

This is one of the most important technical concepts. Follow this chain:

Step-by-Step Data Generation Pipeline

Step 1: Skill Training (Reinforcement Learning)
        ↓
Step 2: Task-and-Motion Planning (TAMP)
        ↓
Step 3: Parallel Simulation Execution
        ↓
Step 4: Labeled Demonstration Dataset

Breaking Down Each Step

Step 1 — Skill Training

• Uses Reinforcement Learning (RL) inside Isaac Lab (Nvidia's simulation framework)
• Robot learns through trial and error
• Learns non-prehensile skills — actions that do not involve picking up objects:
  • Pushing a block across the table
  • Flipping a block onto another face
• Uses privileged information — exact position and orientation values the simulator provides directly, which a real robot cannot access
• This allows automatic skill generation that would otherwise require a human operator

Step 2 — Task-and-Motion Planning (TAMP)

• Automatically selects and sequences the skills needed to solve a task
• Plans the robot's movements between skills
• No human involvement required

Step 3 — Parallel Execution

• Runs hundreds of plans simultaneously on a single GPU
• Achieves up to 24× real-time speed
• Makes large-scale data generation practical

Step 4 — Labeled Dataset

• Each recorded demonstration includes labels describing:
  • Which task the robot is performing
  • Which skill is being executed at each moment

Why This Matters

Traditional data collection requires teleoperation — a human remotely controlling the robot for every demonstration. ToyBlocks eliminates this bottleneck entirely.

---

Step 5: The Two Levels of Robot Intelligence

This is a critical conceptual distinction running throughout the entire article.

High-Level vs. Low-Level Models

HIGH-LEVEL MODEL (Planner)
"What should I do next?"
→ Selects which skill to use
→ Identifies which block to act on
→ Sequences actions across many steps
        ↓ gives instructions to ↓
LOW-LEVEL MODEL (Executor)
"How exactly do I move?"
→ Maps sensor observations to physical actions
→ Controls the robot arm precisely
→ Executes the commanded skill

Concrete Example from ToyBlocks

Why This Separation Is Useful for Research

When a robot fails at a long task, you can now ask:

• Did the planner choose the wrong action? → High-level failure
• Did the executor move incorrectly? → Low-level failure

ToyBlocks provides oracles — reference components with correct plans or actions — at both levels, so you can isolate exactly where failure occurred.

---

Step 6: Studying Generalization

What Generalization Means in Robotics

A robot generalizes when it successfully performs tasks or handles situations it was not explicitly trained on.

Types of Generalization ToyBlocks Tests

Interpolation — New configurations within the training range

Example: Trained on positions in the left half of the table
         Tested on new positions still in the left half

Extrapolation — Configurations beyond the training range

Example: Trained on positions in the left half of the table
         Tested on positions in the right half

Compositional Generalization — Combining familiar skills in new sequences

Example: Trained on stacks of 2 and 5 blocks
         Tested on stacks of 3 and 4 blocks
         (Can the robot combine what it knows?)

Appearance Generalization — New letters, colors, or block appearances not seen during training

Key Finding

Studies on VLA (Vision-Language-Action) generalization have produced conflicting results — some show brittleness, others show strong generalization. ToyBlocks argues this conflict exists because real-world datasets have limited flexibility in spatial variation. ToyBlocks provides the controlled conditions needed to resolve this question properly.

---

Step 7: Agentic AI Systems in ToyBlocks

What an Agentic System Is

An agentic system is an AI that:

• Perceives its environment
• Plans a sequence of actions
• Calls tools to execute those actions
• Adapts based on results

How ToyBlocks Enables Agentic Research

ToyBlocks exposes robot skills through the Model Context Protocol (MCP) — a standard for connecting AI applications to tools.

The agent can:

1. See the scene through camera images
2. Call skills (pick, flip, rotate) by pointing at a pixel
3. Decide when to search, build, or declare a task impossible

Three Experimental Settings

Setting 1 — Classical Planning

• Agent uses text descriptions of the scene
• Solves 30/30 block-stacking tasks
• Also solves 30/30 using camera images in a closed loop

Setting 2 — Search and Spell

• Agent must find blocks with specific letters among up to 8 blocks
• 50% of scenes are infeasible (missing required letters)
• Agent must recognize infeasibility and say so
• Results: 63% success rate
• Main errors:
  • Confusing look-alike letters: 13%
  • Incorrect letter orientation: 15%

Setting 3 — Self-Directed Experimentation

• Agent behaves like a scientist
• When a skill behaves unexpectedly, it:
  1. Forms a hypothesis
  2. Designs a small test
  3. Revises its understanding

Result comparison:
Default skill descriptions:        24% success
Human expert descriptions:         36% success
Agent's own learned descriptions:  63% success

Key Insight: An agent that writes its own skill descriptions through experimentation outperforms both default descriptions and human-written ones.

---

Step 8: World Models as Simulators

What Action-Conditioned World Models Are

These are models trained to predict how a scene changes in response to a robot's actions — essentially learning to simulate physics from data rather than using a hand-coded physics engine.

The Research Question

Can these learned world models replace traditional simulators for training and evaluating robots?

Why ToyBlocks Is Uniquely Useful Here

To test a world model, you need ground truth — the actual state of the scene to compare against predictions. ToyBlocks provides this directly from the simulator.

What Was Tested

• 4 skills: pick-and-place, flip, rotate, mixed task
• 900 trajectories per skill (50% successful, 50% unsuccessful)
• Models must predict whether a skill execution will succeed

Three Model Types Tested

Two Evaluation Modes

• Full trajectories: Errors accumulate over long sequences
• Short interaction segments: Isolates a single interaction (e.g., gripper touching a block)

---

Step 9: Sim-to-Real Transfer

The Core Challenge

A robot trained entirely in simulation must work on a physical robot without any real-world training examples. This is called zero-shot sim-to-real transfer.

The Three-Stage Transfer Process

Stage 1: BEFORE Data Generation
→ System Identification
→ Tune simulated joint friction and motor inertia
   to match real robot trajectories

Stage 2: DURING Data Generation
→ Domain Randomization
→ Vary: object size, friction, table height (physical)
→ Vary: lighting, backgrounds, materials, camera parameters (visual)

Stage 3: AT INFERENCE TIME
→ Use real cameras (ZED front-facing + ZED Mini wrist-mounted)
→ Generate clean depth maps from RGB images using FastFoundationStereo
   (avoids noisy depth sensor measurements)

Results

Notable finding: Simulation performance closely predicted real-world performance, validating ToyBlocks as a meaningful proxy for physical deployment.

---

Step 10: Putting It All Together — The Full Picture

TOYBLOCKS FRAMEWORK OVERVIEW

[Controlled Simulation]
    ↓
[Automatic Skill Training via RL]
    ↓
[Task-and-Motion Planning → Demonstrations]
    ↓
[Parallel Data Generation at Scale]
    ↓
[Training: High-Level (Molmo 2) + Low-Level (ManiFlow)]
    ↓
[Evaluation: Generalization, Planning, Agentic Behavior, World Models]
    ↓
[Sim-to-Real Transfer → Physical Robot]
    ↓
[Metrics: Success rates, execution speed, forces, task progress]

---

Summary Table: Key Concepts

---

Self-Check Questions

1. Why is a narrow, controlled environment more useful than a broad one for studying robot learning?
2. What is the difference between interpolation and extrapolation generalization, and why does the distinction matter?
3. Explain why separating high-level and low-level models helps researchers diagnose failures.
4. Why did the agent's self-written skill descriptions outperform human-written ones?
5. What three stages does ToyBlocks use to bridge the gap between simulation and the real world?
6. If a world model always predicts success, what accuracy would it achieve on ToyBlocks' test set, and why is that the baseline?