Step-by-Step Teaching

STEP 1: The Core Problem — Why Traditional Robots Struggle

Traditional Robot Workflow:
New Task → Collect Data → Retrain Model → Validate → Deploy
         ↑_______________________________________________|
                    (Repeat for EVERY change)

Think of it like this:

Imagine hiring an employee who needs months of retraining every time you rearrange the office furniture

Key limitations:

• Fixed programming = rigid behavior
• New product arrives → significant reprogramming required
• Layout changes → more data collection needed
• Each change = expensive time and engineering resources

---

STEP 2: The Breakthrough Concept — In-Context Learning

What is In-Context Learning?

How S1's In-Context Learning Works:

Step 1: Operator records video of desired task
           ↓
Step 2: Video fed to S1 as a "prompt"
           ↓
Step 3: Model interprets:
        • Demonstrated INTENT
        • OBJECTS involved  
        • ACTION SEQUENCE
           ↓
Step 4: Maps understanding → Robot actions
           ↓
Step 5: Robot executes task (NO retraining needed)

Why This Matters — The Numbers:

1 video demonstration ≈ 380 hands-on training examples
                      ≈ 50-100 hours of manual collection

S1 success rate:    66% per step
Comparable AI:       9% per step
                   ─────────────
Improvement:        7x better

Real example: Plant potting task — from recording demonstration to autonomous robot execution in just 11 minutes

---

STEP 3: Understanding Foundation Models in Robotics

What is a Robot Foundation Model?

Think of it like ChatGPT but for robot actions:

Language Model:          Robot Foundation Model:
Text input               Video/sensor input
↓                        ↓
Understands language     Understands physical tasks
↓                        ↓
Generates text output    Generates robot movements

Key Capabilities of S1:

• ✅ Handles tasks lasting up to 10 minutes
• ✅ Executes dozens of manipulation steps
• ✅ Composes skills in new sequences never seen before
• ✅ Recovers from errors when objects move
• ✅ Works across multiple robot types (embodiments)

Example tasks S1 can learn from one video:

• 🌱 Plant potting
• 🥞 Pancake making
• ☕ Pour-over coffee brewing
• 📦 Kit assembly

---

STEP 4: The AI Development Pipeline

This is the complete journey from idea to factory deployment:

Stage 1: Data Generation

NVIDIA Cosmos (World Foundation Models)
├── Converts video → structured descriptions
├── Diversifies training data
└── Cosmos Curator: annotates, filters, organizes data at scale

Stage 2: Simulation & Training

NVIDIA Omniverse + Isaac Sim
├── Creates physically accurate virtual environments
├── Tests edge cases safely (no real robot damage)
└── Generates synthetic training data

NVIDIA Isaac Lab (Reinforcement Learning)
├── Powered by Newton physics engine
├── Models: forces, contact, collision, pressure
└── Reduces simulation-to-reality gap

Stage 3: Optimization & Deployment

NVIDIA Nsight Tools
└── Finds performance bottlenecks during training

NVIDIA TensorRT SDK
└── Optimizes inference speed for real-time robot response

The Complete Pipeline Visualized:

Raw Data/Video
     ↓
[Cosmos] → Structured, diverse training data
     ↓
[Isaac Sim] → Safe virtual testing environment
     ↓
[Isaac Lab] → Reinforcement learning with physics
     ↓
[Nsight] → Performance optimization
     ↓
[TensorRT] → Fast real-world inference
     ↓
Physical Robot Deployment

Key insight: These aren't separate systems — they're connected stages of one unified pipeline

---

STEP 5: Real-World Application — From Lab to Factory

The Simulation-to-Reality Gap Problem

Problem: Robots trained in simulation often fail in real world
         because physics isn't perfectly modeled

Solution: Newton physics engine accurately models:
          • How robots TOUCH objects
          • How robots GRIP objects  
          • Force and pressure dynamics
          • Collision behavior

Real Deployment Example — NVIDIA Blackwell Assembly:

Task: High-precision electronics assembly
Partners: Skild AI + NVIDIA + Foxconn

Robot performs:
1. Install busbar and limit block
2. Fasten 16 screws precisely
3. Adapt to disturbances mid-task
4. Track sequence across multiple steps
5. Recover when scene differs from plan

Required capabilities:

• Precise motion control
• Contact-aware manipulation
• Sequence memory
• Error recovery

---

STEP 6: Connecting the Concepts — The Big Picture

Why This Represents a Paradigm Shift:

OLD PARADIGM:              NEW PARADIGM:
"Program robots"           "Teach robots"
     ↓                          ↓
Fixed instructions         Learning from experience
     ↓                          ↓
Breaks with change         Adapts to change
     ↓                          ↓
Engineer-dependent         Operator-accessible

The Flywheel Effect:

More deployments
      ↓
More real-world experience data
      ↓
Better model performance
      ↓
Faster future deployments
      ↓
(cycle continues)

Skild reached $100M annual revenue in just 10 months with 60+ deployment partnerships — evidence this approach works commercially

---

Summary: Key Concepts to Remember

---

Self-Check Questions

1. What makes in-context learning different from traditional robot training?
2. Why is simulation important before real-world deployment?
3. What problem does the Newton physics engine solve?
4. How does one video demonstration compare to manual training examples?
5. Explain the development pipeline in your own words from data to deployment