After studying this material, students should be able to:

1. Explain what NVIDIA Omniverse is and how AI agents integrate with it
2. Identify the core Omniverse libraries and their specific functions
3. Describe the developer workflow for building simulations using frontier AI agents
4. Recognize real-world use cases where AI-assisted simulation applies
5. Understand how natural language instructions direct AI agents to assemble simulation components

---

Step-by-Step Teaching

Step 1: Understanding the Core Problem Being Solved

Before diving into tools, understand why this matters.

Building a simulation traditionally requires:

• Writing complex code manually
• Deep expertise in physics engines, rendering, and 3D assets
• Significant time connecting multiple systems together

The challenge looks like this:

Idea → Assets → Physics → Rendering → Sensors → Testing → Working App
         ↑          ↑          ↑           ↑          ↑
    (each step requires specialized knowledge and code)

The solution introduced here:

Developers use natural language instructions to direct frontier AI agents (like GPT-6 Astra or Claude Fable 5), which then generate the code and connections needed — while the developer reviews and guides the results.

---

Step 2: Understanding the Key Players

2A: What is NVIDIA Omniverse?

Think of Omniverse as a platform of building blocks for simulation. It provides:

2B: What are Frontier AI Agents?

These are advanced AI models capable of:

• Understanding natural language instructions
• Writing and executing code
• Connecting multiple software libraries
• Iterating based on feedback

Examples mentioned:

• GPT-6 Astra — used across most projects
• Claude Fable 5 — used alongside Astra for sensor validation

2C: What is OpenUSD?

Universal Scene Description (USD) is a file format and framework for describing 3D scenes. Think of it as the common language all tools speak when sharing 3D data.

---

Step 3: Learning the Core Omniverse Libraries

These libraries are the building blocks AI agents connect together:

┌─────────────────────────────────────────────────────┐
│              OMNIVERSE LIBRARY STACK                │
├──────────────┬──────────────────────────────────────┤
│  ovphysx     │  Physics simulation                  │
│  ovstage     │  Scene updates and management        │
│  ovrtx       │  Rendering (visual output)           │
│  ovui        │  User interface                      │
│  ovstream    │  Streaming to browser/other apps     │
│  simready    │  Pre-built simulation-ready assets   │
└──────────────┴──────────────────────────────────────┘

Key insight: The AI agent's job is largely to connect these libraries correctly based on what the developer needs.

---

Step 4: Understanding the Developer Workflow

This is the core process used across all projects:

Step 1: Developer has an idea
           ↓
Step 2: Developer gives natural language instruction to AI agent
           ↓
Step 3: AI agent maps out the workflow / selects libraries
           ↓
Step 4: AI agent generates code connecting the components
           ↓
Step 5: Developer reviews results
           ↓
Step 6: Developer guides corrections or refinements
           ↓
Step 7: Working simulation application

Important: The developer does not disappear. They remain in control — directing, reviewing, and correcting at each stage.

---

Step 5: Learning Through Real Use Cases

Now apply the framework above to each project. Notice the pattern repeating in every case.

---

Use Case 1: Humanoid Warehouse Simulator

Problem: Need an interactive simulation to test warehouse robot behavior before real deployment

What the developer did:

• Directed Astra to connect: ovphysx + ovstage + ovrtx + ovui + simready
• Astra generated animation and application code

Result: Interactive simulator with first- and third-person views

Concept learned:

Multiple Omniverse libraries can be combined by an AI agent to create a complete interactive environment from a natural language request

---

Use Case 2: Autonomous Driving Testing

Problem: Scene, sensor, or model changes can unpredictably affect self-driving simulations

What the developer did:

• Asked Astra to build a reusable simulation of San Francisco's Market Street
• Connected: asset creation → traffic → RTX sensor simulation → driving model in stages
• Used Cosmos3-Nano to vary weather/lighting in recorded videos

Result: A testing ground to compare how changes affect driving behavior

Concept learned:

AI agents can build staged, modular workflows — connecting components one at a time and checking each integration before moving forward

---

Use Case 3: Digital Twin Sensor Validation

Problem: Simulated sensors must closely match real sensors for reliable robot/vehicle testing

What the developer did:

• Directed two AI agents (Astra + Claude Fable 5) to compare:
  • ovrtx camera output vs. recorded camera data
  • Raw LiDAR output vs. recorded LiDAR data
• Agents measured differences, modified OpenUSD scenes, checked results
• Process ran iteratively over ~3 days

Result: Created 2 digital twins from scratch, improved 2 existing ones

Concept learned:

AI agents can run iterative improvement loops — measure → modify → check → repeat — guided by quantitative metrics (KPIs)

New term — Digital Twin:

A digital twin is a virtual replica of a real physical object or environment, accurate enough to test behavior and predict outcomes

---

Use Case 4: Robo Olympics

Problem: Need to verify that robot movements work under real physical constraints

What the developer did:

• Used sports videos + natural language to guide Astra
• Astra built controllers and refined them through physics trials
• Tools used: Newton Physics Engine + NVIDIA Warp (open source) + ovrtx

Result: Robot cleared a hurdle in 64 of 100 simulation trials — giving data to improve timing and control

Concept learned:

Simulation trials generate quantitative feedback (e.g., 64/100 success rate) that developers use to improve robot behavior iteratively

---

Use Case 5: Robotic Disassembly Testing

Problem: Before deploying a robot to disassemble products, verify its tools can physically reach and remove parts

What the developer did:

• Directed Astra to model a car suspension in PTC Onshape (CAD tool)
• Configured it in NVIDIA Isaac Sim
• Astra measured available space and designed a custom wrench for the robot

Result: Successful removal of a suspension component in simulation

Concept learned:

AI agents can bridge design tools (CAD) and simulation tools, connecting design decisions directly to physical testing outcomes

---

Use Case 6: International Space Station Browser App

Problem: Turning 3D models + live data into a browser-accessible application

What the developer did:

• Used a single prompt to assemble NASA assets into an OpenUSD ISS model with telemetry
• Used a follow-up prompt to adjust the scene (shift to Earth's daytime side)
• Tools: Blender (asset prep) + ovrtx + ovstage + ovstream

Result: 3D ISS model with operational data, viewable in a browser

Concept learned:

Complex multi-component applications can be initiated with a single natural language prompt, with refinements made through follow-up prompts

---

Use Case 7: Room Capture to Testing Environment

Problem: A scanned room needs editable objects and accurate physics before interaction testing

What the developer did:

• Used stereo camera captures processed through: PyCuSFM + FoundationStereo + nvblox
• Astra assembled assets and used USD Content Agents to configure object behavior
• Isaac Sim tests guided collision and contact revisions

Result: Editable studio environment where doors and drawers behave physically correctly

Concept learned:

Real-world captured data (camera scans) can be transformed into editable, physics-accurate simulations through an AI-guided pipeline

---

Step 6: Synthesizing the Big Picture

Now connect everything into one unified understanding:

┌─────────────────────────────────────────────────────────────┐
│                    THE OMNIVERSE AI WORKFLOW                │
│                                                             │
│  DEVELOPER (idea + direction + review)                      │
│       ↕  natural language                                   │
│  AI AGENT (Astra / Claude Fable 5)                          │
│       ↕  generates & connects code                         │
│  OMNIVERSE LIBRARIES (physics, rendering, sensors, scene)   │
│       ↕  outputs                                            │
│  SIMULATION (interactive, testable, improvable)             │
└─────────────────────────────────────────────────────────────┘

Three Roles Always Present:

---

Step 7: Key Concepts Summary

---

Quick Self-Check Questions

1. What is the developer's role when using AI agents for simulation — are they passive or active?
2. Name three Omniverse libraries and what each one does
3. What does "iterative workflow" mean in the context of digital twin improvement?
4. How did the autonomous driving project use staged integration?
5. What is the difference between a digital twin and a regular 3D model?
6. Why would a developer simulate robotic disassembly before doing it physically?

---

Core takeaway: Frontier AI agents lower the barrier to building complex simulations by translating natural language into working code that connects specialized libraries — but the developer remains essential as the director who defines goals, reviews outputs, and guides improvements.