Step-by-Step Study Material

STEP 1: Understanding the Problem

The Core Challenge

1% of all live births → congenital heart defect
Every defect is UNIQUE
Every child's anatomy is DIFFERENT

Why This Is Difficult

Real-World Consequence

A child underwent two failed heart repair surgeries because surgeons could not locate the defect using traditional methods

Key Insight: Surgeons needed to understand the exact anatomy before entering the operating room

---

STEP 2: The Solution Framework — Three Core Needs

Before learning the tools, understand what problems needed solving:

NEED 1: See the heart accurately
         ↓
NEED 2: Model it quickly enough for clinical use
         ↓
NEED 3: Simulate what happens BEFORE surgery

---

STEP 3: The Technology Stack — Layer by Layer

Layer 1 — Medical Imaging Input

Sources of data already available to care teams:

• CT scans
• MRI
• 3D Ultrasound (echocardiography)

These images existed before AI — the challenge was processing them fast enough and precisely enough

---

Layer 2 — 3D Slicer & SlicerHeart

What it is:

• 3D Slicer = open source software for visualizing and analyzing 3D medical images
• SlicerHeart = an extension built specifically for pediatric hearts

What it does:

Raw medical images → Segmented, visualized 3D heart model

Who built it: Dr. Matthew Jolley's lab at CHOP, with open source community contributions from Stanford and Boston Children's Hospital

---

Layer 3 — MONAI + Auto3DSeg (The AI Acceleration Layer)

What is MONAI?

• Open source medical imaging AI framework
• Co-founded by NVIDIA

What is Auto3DSeg?

• NVIDIA's automated 3D segmentation implementation within MONAI
• Trained on pairs of prior images and manually created models

The Machine Learning Workflow:

Step 1: Collect 10–20 image-model pairs (human-made)
Step 2: Train a segmentation network on those pairs
Step 3: Apply the trained model to new patient images
Step 4: Output meets same quality as human expert

The Impact:

"Machine learning has become just bread and butter" — Dr. Jolley

---

Layer 4 — Newton & NVIDIA Warp (The Simulation Layer)

What is NVIDIA Warp?

• A Python framework that runs physics simulations on GPUs

What is Newton?

• An open source physics engine built on Warp
• Originally designed for robot training simulations
• Now being adapted for biomechanical cardiac simulation

What does this layer solve?

Visualization answers: "What does the heart look like?"

Simulation answers: "What will HAPPEN when we put a device inside it?"

3D Heart Model + Device Specifications
            ↓
     Newton/Warp Simulation
            ↓
   Predicted device behavior in THIS child's anatomy

Time improvement:

Clinical Application:

• Closure devices (sealing holes in hearts)
• Transcatheter valve simulations (future goal)
• Same-day decision making becomes possible

---

Layer 5 — OpenUSD & NVIDIA Omniverse (The Interaction Layer)

What is OpenUSD?

• Open source 3D interoperability standard
• Allows diverse data and solvers to be combined

What is NVIDIA Omniverse?

• Digital twin platform

What this enables:

Patient Images → 3D Simulation → Virtual Reality Environment
                                          ↓
                          Clinician interacts using natural language
                          (Vision-Language Models / VLMs)
                                          ↓
                          "Show me how Device A fits vs Device B"

This layer is currently in development — it represents the future direction

---

STEP 4: The Complete Workflow — End to End

PATIENT ARRIVES
      ↓
Existing images collected (CT, MRI, Ultrasound)
      ↓
MONAI + Auto3DSeg segments the anatomy (seconds)
      ↓
SlicerHeart generates precise 3D heart model
      ↓
Newton/Warp simulates device deployment
      ↓
Clinician compares device options in near real-time
      ↓ (future)
VR environment with language-model interaction
      ↓
INFORMED SURGICAL DECISION — before anyone enters the OR

---

STEP 5: Why Open Source Is the Critical Enabler

The Economics Problem

~2.4 million people in U.S. have congenital heart disease
Each case is unique → no standardized solution
Small market → traditional companies won't invest
No single institution can build everything alone

How Open Source Solves This

The Collaboration Network

• CHOP — core development, clinical application
• Boston Children's Hospital — 500+ modeled surgeries/year, tool contributions
• Stanford — additional tool development
• National consortium — forming to build next-generation shared infrastructure
• NVIDIA — maintains industrial-scale open platforms

"Open source defies traditional economics for small and heterogeneous populations" — Dr. Jolley

---

STEP 6: Measuring Real-World Impact

Current Scale

Clinical Milestones

• Ventricular septal defect (hole between heart chambers) modeling is now standard of care at CHOP
• The failed-repair case: 3D modeling led to first-attempt success after two prior failures
• CHOP's IDEA Lab is expanding these tools beyond cardiology to multiple disciplines

---

Concept Summary Map

CONGENITAL HEART DISEASE (unique per child)
              ↓
         IMAGING DATA
              ↓
    MONAI + Auto3DSeg (AI segmentation)
              ↓
    SlicerHeart / 3D Slicer (3D modeling)
              ↓
    Newton + NVIDIA Warp (physics simulation)
              ↓
    OpenUSD + Omniverse + VLMs (interaction)
              ↓
    SAFER, MORE PRECISE PEDIATRIC CARDIAC CARE
              ↑
    Enabled throughout by OPEN SOURCE collaboration

---

Key Terms to Remember

---

Self-Check Questions

1. Why can't surgeons simply use off-the-shelf devices for children with congenital heart defects?
2. What was the workflow before AI, and what made it impractical for routine clinical use?
3. What specific role does MONAI play versus Newton in this system?
4. Why is open source particularly important for rare pediatric conditions?
5. What is the difference between visualization and simulation in this context, and why does that distinction matter clinically?