How Agent Lightning Trains Real AI Agents with RL

🎯 Learning Outcomes

After studying this material, you should be able to:

1. Explain the limitations of traditional agentic RL frameworks
2. Define Harnessed Agentic RL and how it differs from traditional agentic RL
3. Identify the four key technical challenges in training with real harnesses
4. Describe the three core components of Agent Lightning v1.0's architecture
5. Explain Collocated Async RL and its efficiency advantages
6. Evaluate the practical results achieved using Agent Lightning v1.0

---

📚 Step-by-Step Study Material

---

STEP 1: Understanding the Problem — Why Traditional Agentic RL Falls Short

What is Agentic RL?

Reinforcement Learning (RL) teaches AI systems through trial and error:

• The agent takes an action
• The environment returns an observation
• The agent receives a reward or penalty
• Repeat until the agent improves

Traditional RL Loop:
Model → Action → Environment → Observation → back to Model

What is an Agent Harness?

A harness is the external system that coordinates the AI model with tools and environments. Think of it as the "wrapper" around the model that handles:

• Context management
• Tool protocols
• Execution logic
• Dependencies

Real-world examples: mini-SWE-agent, OpenHands, Claude Code, Codex

The Core Problem

Traditional RL frameworks own the interaction loop — meaning they control every step between the model and environment.

Key Insight: You train one agent but deploy a different one. This gap undermines the entire purpose of training.

---

STEP 2: The Solution — Harnessed Agentic RL

Core Idea

Instead of rebuilding the agent inside the training framework, place an LLM proxy between the agent and the model.

Traditional Agentic RL:
Environment ↔ [Tokenizer] ↔ Policy Model

Harnessed Agentic RL:
Environment ↔ [Agent Harness] ↔ [OpenAI-like API Proxy] ↔ Policy Model

How It Works

1. The agent harness runs exactly as it does in deployment
2. Instead of calling the real model API, it calls the Agent Lightning proxy
3. The proxy records all prompts, responses, and log probabilities
4. Training happens using those recorded interactions

Practical implication: To connect an existing agent to RL training, you only need to change the API endpoint URL. No code rewriting required.

Why This Matters

---

STEP 3: Four Technical Challenges (and Why They Exist)

Because the harness controls the interaction loop, the training system only sees a series of LLM request/response pairs — not a clean, continuous token trajectory. This creates four specific problems:

---

Challenge 1: Retokenization and Sample Merging

The problem:

• Harnesses store context as text
• RL training needs token IDs from the original rollout
• Re-running text through the tokenizer can shift token boundaries
• Adjacent calls cannot always be safely merged into one sample

Analogy: Imagine recording a conversation by summarizing it, then trying to reconstruct the exact words. The meaning may be preserved, but the precise wording changes.

---

Challenge 2: Advantage Calculation

The problem:

• One rollout may be split into multiple samples due to retokenization, subagents, or context summarization
• Computing advantages at the sample level means rollouts with more samples get counted more times
• This distorts the statistical relationships that RL depends on

Correct approach: Calculate advantages at the rollout level, not the sample level.

Wrong:  Advantage per sample → rollouts with 5 samples weighted 5x more
Right:  Advantage per rollout → each rollout weighted equally

---

Challenge 3: Loss Normalization

The problem:

• Averaging loss by sample count gives more weight to rollouts that produce more samples
• Sample count is often just a product of harness behavior, not task difficulty
• This introduces unintended bias into training

Correct approach: Normalize loss at the rollout level to avoid this distortion.

---

Challenge 4: Training Backend Scheduling

The problem:

• Sample count and length are only known after the harness finishes
• GPU counts and parallel configurations are fixed in advance
• The system must map a variable workload into fixed resources

This requires dynamic scheduling logic that can handle unpredictable batch sizes.

---

STEP 4: System Architecture — Three Core Components

Agent Lightning v1.0 is built from ~3,500 lines of code organized into three components:

┌─────────────────────────────────────────────────────┐
│              AGENT LIGHTNING v1.0                   │
│                                                     │
│  [Agent Harnesses]  →  [API Gateway]                │
│  mini-SWE-agent         Rollout API                 │
│  OpenHands              LLM API Proxy               │
│  OpenClaw                    ↓                      │
│                        [Rollout Controller]         │
│                         Local Reconciler            │
│                         Kubernetes Reconciler       │
│                              ↓                      │
│                        [Customized Trainer]         │
│                         Sample Adapter              │
│                         Monitoring                  │
│                              ↓                      │
│                    [Inference + Training Engine]    │
└─────────────────────────────────────────────────────┘

Component 1: API Gateway

Role: The central hub and LLM proxy

Functions:

• Acts as an OpenAI-compatible API so harnesses need no modification
• Links every model call to its rollout ID
• Records prompts, responses, and log probabilities needed for training
• Stores rollouts, models, and events

---

Component 2: Rollout Controller

Role: Manages agent execution

Functions:

• Starts and monitors agent jobs
• Supports two execution modes:
  • Local processes (for small-scale or development use)
  • Kubernetes jobs (for large-scale production use)
• Keeps agent execution separate from the trainer for clean modularity

---

Component 3: Customized Trainer

Role: Orchestrates the training process

Functions:

• Built on top of verl (an existing RL training library)
• Creates rollouts and waits for completion
• Collects samples from the API Gateway
• Assembles final training samples through a sample adapter
• Handles the rollout-level advantage and normalization logic

---

STEP 5: Collocated Async RL — Solving the GPU Efficiency Problem

The GPU Scheduling Challenge

Running many agents simultaneously is resource-intensive. There are three approaches:

How Collocated Async RL Works

Timeline:
[Rollout] [Rollout] [UPDATE] [Rollout] [Rollout] [UPDATE]
    ↑           ↑        ↑
 Agents      Agents   GPUs switch to training
 running     running  (API Gateway pauses new requests)

Step-by-step process:

1. Agents run rollouts on shared GPUs
2. Once enough rollouts are collected, training update begins
3. API Gateway pauses accepting new requests
4. Gateway waits for in-progress requests to finish
5. Model update completes
6. Rollout resumes — harness sees no interruption

Result: ~2x end-to-end speedup over synchronous RL, using fewer GPUs than asynchronous RL

---

STEP 6: Kubernetes Integration — Scaling Without Commercial Services

The Scaling Problem

Running many agents requires substantial CPU, memory, and compute. Other frameworks solve this by using paid commercial sandbox services (Modal Sandbox, E2B), which become expensive at scale.

Agent Lightning's Approach

Agents run as standard Kubernetes jobs on:

• Self-managed clusters
• Cloud Kubernetes (AKS, EKS, GKE)
• Local infrastructure

Flow:
API Gateway (holds rollout queue)
      ↓ polls
Rollout Controller
      ↓ creates
Kubernetes Jobs (agents run here)
      ↓ status updates
API Gateway

Benefits:

• Reuses existing infrastructure — no new costs
• Fully open source and reproducible
• Scales to large rollout batches efficiently

---

STEP 7: Experimental Results — Validating the Approach

The Experiment Setup

Results

Before RL training:  41.8% Pass@1
After RL training:   56.4% Pass@1
                    ─────────────
Improvement:        +14.6 percentage points

Significance: A 14.6-point gain using only ~6,000 samples and no large-scale compute demonstrates that Harnessed Agentic RL is both effective and accessible.

Confirming the Technical Choices

The experiments validated the advantage calculation and loss normalization decisions:

Takeaway: Both rollout-level advantage AND rollout-level normalization are necessary together for best results.

---

🔑 Key Concepts Summary

---

✅ Self-Check Questions

1. Why does traditional agentic RL require reimplementing the agent inside the training framework?
2. What single change does a developer need to make to connect an existing harness to Agent Lightning?
3. Why can't adjacent LLM calls always be merged into one training sample?
4. What happens to GPU utilization in synchronous RL, and how does Collocated Async RL fix it?
5. Why is rollout-level normalization preferred over sample-level normalization?
6. What was the Pass@1 improvement on SWE-bench Verified, and why is the sample count significant?