Step-by-Step Study Material

Step 1: Understand the Problem First (Why AIR Exists)

Before learning the solution, understand the three core challenges AI research teams face:

💡 Key Insight: Without proper infrastructure, researchers waste time fixing broken GPUs, reconfiguring environments, and manually coordinating resources — instead of doing actual research.

---

Step 2: Learn the Core Features of AIR

🔧 Feature 1: Built-in GPU Resiliency

What it does:

• Runs preflight health checks on GPU nodes before training starts
• Monitors nodes during training for problems like:
  • NCCL communication failures
  • Thermal/power throttling
  • GPU driver issues
• Automatically removes unhealthy nodes so future jobs avoid them

Why it matters:

Without AIR → GPU fails → researcher manually investigates → hours lost
With AIR    → GPU fails → AIR detects & removes it → next job runs cleanly

📌 Real Example: Training models with 100+ billion parameters (like KARL) requires thousands of GPU hours. One infrastructure failure without resiliency wastes enormous time and money.

---

🚀 Feature 2: Fast Iteration with AIR CLI and Custom Containers

Two components work together:

AIR CLI (Command Line Interface):

• Launch workloads programmatically (no manual clicking)
• Specify exact GPU requirements
• AIR handles provisioning, scheduling, and high-bandwidth networking automatically

Custom Docker Containers:

• Researchers build environments locally
• Same container image runs in training — no compatibility surprises
• Images stored and shared via Databricks Artifact Registry
• Jobs start in seconds, not minutes

💡 Why seconds matter: When testing a new idea, waiting 10 minutes to start a job kills momentum. Fast launches = faster learning cycles.

---

📊 Feature 3: MLflow Integration for Experiment Tracking

What happens automatically:

1. AIR creates and manages MLflow experiments for each training job
2. Researchers can monitor job status in real time
3. Training logs are streamed and inspectable
4. Metrics are retrievable to compare experiments

Why this matters:

• No separate configuration needed
• All experiment history is organized and searchable
• AI agents can also read these metrics (important for Step 4)

---

🗄️ Feature 4: Governed Data Access via Unity Catalog

The problem it solves:

• Research data lives in Unity Catalog (Databricks' data governance layer)
• Training jobs run across different workspaces and clouds
• Data needs to be accessible, tracked, and secure everywhere

How AIR solves it:

• Unity Catalog FUSE mounting makes remote data appear as a local filesystem
• Training code doesn't need to change — it reads files normally
• Data access is automatically tracked and governed

Training Code sees:  /data/documents/file.pdf  ← looks local
Actually stored in:  Unity Catalog             ← governed & secure

---

⚖️ Feature 5: GPU Resource Management and Prioritization

The challenge:

• Small debug job: needs 2 GPUs for 1 hour
• Large training run: needs 500 GPUs for 2 weeks
• Both projects have deadlines — who gets resources?

AIR's solution:

• Workload prioritization: Time-sensitive experiments get resources first
• Preemption: Lower-priority jobs pause when urgent jobs need GPUs
• Spare capacity use: Idle GPUs automatically run exploratory experiments (e.g., over weekends)

📌 Result: Researchers don't manually negotiate for GPU time — AIR handles allocation automatically.

---

Step 3: Understand the Shift to Agent-Driven Research (Autoresearch)

What is Autoresearch?

Instead of researchers manually running every experiment:

Traditional:  Researcher → designs experiment → runs it → evaluates → repeat
Autoresearch: Agent → proposes change → runs experiment → reads results → iterates

Impact: Training job volume grew 8x because agents run experiments continuously.

The Autoresearch Cycle (4 Steps)

1. PROPOSE  → Agent suggests a model change or hyperparameter
       ↓
2. RUN      → Agent launches training job via AIR CLI
       ↓
3. EVALUATE → Agent reads MLflow metrics from completed job
       ↓
4. ITERATE  → Agent decides what to try next
       ↑_____________________________________________|

Why AIR Is Essential for Autoresearch

💡 Key Insight: A problem that's annoying once (slow startup, confusing error) becomes a critical bottleneck when agents repeat it hundreds of times.

---

Step 4: Connect Everything — The Complete Picture

┌─────────────────────────────────────────────────────┐
│              AI Research Workflow with AIR           │
│                                                     │
│  Data (Unity Catalog) ──→ Training Job              │
│                              ↑                      │
│  Code (Docker Container) ────┤                      │
│                              │                      │
│  Launch (AIR CLI) ───────────┤                      │
│                              │                      │
│  GPU Nodes (H100/B300) ──────┘                      │
│       ↓ (health monitored)                          │
│  Results → MLflow → Researcher or Agent             │
│                         ↓                           │
│              Next Experiment Launched               │
└─────────────────────────────────────────────────────┘

---

Step 5: Key Takeaways to Remember

---

Quick Self-Check Questions

1. What three challenges does AIR address for AI research teams?
2. What happens when AIR detects an unhealthy GPU node during training?
3. Why does job startup speed matter more in agent-driven research than in manual research?
4. How does Unity Catalog FUSE mounting simplify training code?
5. How does AIR handle competing GPU demands from different research projects?