After studying this material, you should be able to:
- Explain the limitations of current UI navigation approaches (frontier models, SFT, offline RL)
- Understand the architecture of a Planning-Grounding agent framework
- Describe how dense reinforcement learning differs from terminal-only RL
- Explain PPO and the novel GiGPPO method conceptually
- Evaluate why online RL outperforms offline alternatives for UI navigation
Step-by-Step Teaching
Step 1: The Problem Space — Why UI Navigation Is Hard
What is UI Navigation?
UI navigation means an AI agent autonomously interacts with a user interface (like an iPhone app) to complete tasks.
Example task: "Open Settings and turn on Dark Mode"
The agent must:
- See the screen (vision)
- Understand the task (language)
- Decide what to tap/type (action)
- Recover if something goes wrong (error handling)
The Three Existing Approaches and Their Failures
┌─────────────────────────────────────────────────────────────┐
│ CURRENT APPROACHES & THEIR PROBLEMS │
├──────────────────┬──────────────────┬───────────────────────┤
│ Frontier VLMs │ SFT Models │ Offline RL │
│ (GPT-4, Gemini) │ │ │
├──────────────────┼──────────────────┼───────────────────────┤
│ ✅ Very capable │ ✅ Cheap │ ✅ Cheap │
│ ❌ Very expensive│ ❌ Memorizes │ ❌ Static data │
│ ❌ Can't use │ fixed paths │ ❌ Can't adapt │
│ every day │ ❌ Brittle │ to new situations │
└──────────────────┴──────────────────┴───────────────────────┘
Key Insight: Think of SFT like a student who memorizes exam answers. They fail when questions change slightly. We need a student who understands the material.
Step 2: The Proposed Solution — Online RL Framework
What Makes This "Online"?
| Offline RL | Online RL (This Paper) | |
|---|---|---|
| Data source | Pre-collected static dataset | Live interactions with real apps |
| Learning | Fixed, no new experiences | Continuously learns from new situations |
| Adaptability | Poor | Strong |
| Analogy | Learning to drive from videos | Learning to drive in an actual car |
Online RL trains directly on hundreds of live iOS devices simultaneously.
The Core Architecture: Decoupled Agent Design
The framework splits one complex agent into two specialized components:
┌─────────────────────────────────────────────────────────────┐
│ TASK: "Book a flight" │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PLANNING AGENT (LLM-based) │
│ │
│ • Understands the high-level goal │
│ • Decides WHAT to do next │
│ • Output: "Tap the search bar" │
│ │
│ Analogy: The navigator in a car giving directions │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ UI GROUNDING MODEL (Vision-based) │
│ │
│ • Looks at the actual screen │
│ • Decides WHERE exactly to act │
│ • Output: Coordinates (x=245, y=380) → tap here │
│ │
│ Analogy: The driver executing the turn │
└─────────────────────────────────────────────────────────────┘
Why decouple?
- Each component can specialize and be optimized independently
- Easier to debug (planning error vs. grounding error)
- More efficient than one monolithic model doing everything
Step 3: The Reward System — Teaching the Agent What "Good" Means
The Problem with Simple Rewards
Imagine teaching a child to clean their room with only one rule:
"You get a reward only when the room is perfectly clean"
Problems:
- No feedback during the process
- Child doesn't know if they're making progress
- Very slow learning
This is terminal-only reward — the agent only learns at the end of a task.
Dense Rewards: Step-by-Step Feedback
This paper uses dense rewards — feedback at every single step:
Task: "Send an email to John"
Step 1: Open Mail app
→ Reward: +0.3 (Good progress!)
Step 2: Tap Compose button
→ Reward: +0.4 (Getting closer!)
Step 3: Type wrong email address
→ Reward: -0.2 (Wrong direction!)
Step 4: Correct the address
→ Reward: +0.3 (Recovery!)
Step 5: Send email successfully
→ Terminal Reward: +1.0 (Task complete!)
Total learning signal = Dense step rewards + Terminal reward
LLM-as-a-Judge: Who Gives the Rewards?
A Large Language Model evaluates each step and assigns rewards:
┌─────────────────────────────────────────────────────────────┐
│ LLM-AS-A-JUDGE │
│ │
│ Input: Screenshot + Action taken + Task description │
│ │
│ Question it answers: │
│ "Did this action make progress toward the goal?" │
│ │
│ Output: Reward score (e.g., 0.0 to 1.0) │
└─────────────────────────────────────────────────────────────┘
This is powerful because:
- No need for humans to manually label every step
- LLM understands context and nuance
- Scalable to thousands of interactions
Step 4: The Optimization Algorithm — PPO and GiGPPO
Understanding PPO (Proximal Policy Optimization)
PPO is a reinforcement learning algorithm. Let's build intuition:
The Core Problem PPO Solves: When updating an AI's behavior, if you change it too drastically, performance can collapse catastrophically.
Without PPO: With PPO:
Policy update too large → Policy update constrained
Performance crashes → Stable improvement
Agent "forgets" everything → Gradual, safe learning
PPO's Key Idea: "Improve, but not too much at once"
It uses a clipping mechanism to prevent updates that are too large:
If new policy is much better than old → clip the update
If new policy is much worse than old → clip the update
Result: Stable, consistent improvement
Advantage Estimation: Measuring "How Good Was That Action?"
PPO needs to know: "Was this action better or worse than expected?"
This is called advantage:
Advantage = Actual Reward - Expected Reward (Baseline)
Positive advantage → Action was BETTER than expected → Do more of this
Negative advantage → Action was WORSE than expected → Do less of this
This paper uses step-level advantage estimation — calculated at each individual step, not just at the end.
GiGPPO: The Novel Contribution
The Problem with Standard PPO Critics:
In standard PPO, a "critic" network estimates expected rewards. But in UI navigation:
- Each screen state is unique
- The critic sees each state only once
- This makes critic estimates unstable and noisy
Standard PPO Critic Problem:
State: [Screenshot of Settings page]
Critic sees this state: 1 time
Estimate quality: Poor (not enough data)
GiGPPO's Solution: Learn from Visually Similar States
GiGPPO = Group in Group PPO
┌─────────────────────────────────────────────────────────────┐
│ GiGPPO INTUITION │
│ │
│ Current state: Settings page (agent A, trajectory 1) │
│ │
│ Find similar states across ALL concurrent trajectories: │
│ • Settings page (agent B, trajectory 2) → reward: 0.8 │
│ • Settings page (agent C, trajectory 3) → reward: 0.7 │
│ • Settings page (agent D, trajectory 4) → reward: 0.9 │
│ │
│ Use ALL these rewards to estimate value for agent A │
│ Result: Much more stable, accurate estimate! │
└─────────────────────────────────────────────────────────────┘
Analogy: Instead of one student estimating how hard an exam is, you ask 100 students who took similar exams. The average is far more reliable.
Standard PPO: Value(state) = f(single trajectory)
GiGPPO: Value(state) = f(all similar states across trajectories)
↑ More stable, less noisy
Step 5: Infrastructure — Making It Work at Scale
The Engineering Challenge
Training online RL requires:
- Agent takes action → App responds → Agent observes → Repeat
- This must happen hundreds of times simultaneously
┌─────────────────────────────────────────────────────────────┐
│ PARALLEL TRAINING INFRASTRUCTURE │
│ │
│ Virtual iOS Device 1 ──┐ │
│ Virtual iOS Device 2 ──┤ │
│ Virtual iOS Device 3 ──┼──→ Central Training System │
│ ... ──┤ (PPO/GiGPPO) │
│ Virtual iOS Device N ──┘ │
│ │
│ N = Hundreds of concurrent devices │
└─────────────────────────────────────────────────────────────┘
Why hundreds?
- More diverse experiences = better generalization
- Faster data collection = faster training
- GiGPPO specifically benefits from concurrent trajectories
Step 6: Results — How Well Does It Work?
Benchmark Performance Summary
Task: UI Navigation on iOS/Android Apps
┌──────────────────────────────┬─────────────────┐
│ Method │ Performance │
├──────────────────────────────┼─────────────────┤
│ SFT Baseline (8B model) │ ████░░░░░░ 1x │
│ Offline RL │ █████░░░░░ ~2x │
│ Terminal-only RL │ ██████░░░░ ~3x │
│ This Paper (Online RL/GiGPPO)│ ██████████ ~7x │
├──────────────────────────────┼─────────────────┤
│ Gemini-3.5-Flash (frontier) │ ████████░░ │
│ Gemini-3.1-Pro (frontier) │ █████████░ │
│ This Paper (8B model) │ █████████░ ≈Pro│
└──────────────────────────────┴─────────────────┘
Key Findings
1. Out-of-Distribution Generalization
Trained on iOS → Tested on Android → 7x improvement over SFT
This shows the agent learned general UI understanding, not just iOS-specific memorization.
2. Competitive with Frontier Models
8B parameter model ≈ Gemini-3.1-Pro performance
Much smaller model, much lower inference cost, comparable performance.
3. Faster Training Convergence
Dense rewards help the agent learn faster than terminal-only rewards
Step 7: Connecting All Concepts Together
┌─────────────────────────────────────────────────────────────┐
│ COMPLETE SYSTEM OVERVIEW │
│ │
│ Live iOS Apps (hundreds of virtual devices) │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌──────────────────┐ │
│ │ Planning │───▶│ UI Grounding │──▶ Action │
│ │ Agent │ │ Model │ on Screen │
│ └─────────────┘ └──────────────────┘ │
│ │ │ │
│ └────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ LLM-as-a-Judge │──▶ Dense Step Rewards │
│ └──────────────────┘ + Terminal Reward │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ GiGPPO Training │──▶ Updated Agent Policy │
│ │ (PPO + neighbor │ │
│ │ conditioning) │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Summary: Key Concepts to Remember
| Concept | What It Is | Why It Matters |
|---|---|---|
| Online RL | Learning from live app interactions | Enables adaptation, error recovery |
| Planning-Grounding Decoupling | Separate WHAT and WHERE components | Specialization, efficiency |
| Dense Rewards | Step-by-step feedback | Faster, more stable learning |
| LLM-as-a-Judge | LLM evaluates each step | Scalable reward without human labels |
| PPO | Constrained policy optimization | Stable training |
| GiGPPO | Neighbor-conditioned value estimation | Better critic stability |
| Parallel Infrastructure | Hundreds of concurrent devices | Scale for online training |
Self-Check Questions
- Why does SFT lead to "brittle execution"?
- What is the difference between a dense reward and a terminal reward?
- Why does GiGPPO use visually similar states from other trajectories?
- What does the Planning agent do vs. the Grounding model?
- Why is the 7x improvement on Android significant if the model was trained on iOS?