A Vision-Language Model is an AI system that can both see images and understand/generate text.
Input: [Image] + [Text Question]
Output: [Text Answer]
Example:
Image: A photo of a kitchen
Text: "What objects are on the counter?"
Output: "A coffee maker, some plates, and a knife block"
Traditional AI models handled vision OR language separately. VLMs combine both, enabling richer understanding of the world.
A visual agent doesn't just answer questions — it acts in an environment over multiple steps.
Single-turn VLM (old):
See image → Answer question → Done
Visual Agent (new):
Step 1: See image → Reason → Take action
Step 2: See new image → Reason → Take action
Step 3: See new image → Reason → Take action
... (continues until goal is reached)
| Agent Type | What It Sees | What It Does |
|---|---|---|
| Web browser assistant | Screenshots | Clicks, types, navigates |
| Household robot | Camera feed | Picks up, moves objects |
| Game agent | Game screen | Moves, interacts |
Each step only gives the agent a partial snapshot — like trying to navigate a city by only seeing one block at a time.
Imagine you're in a house with a blindfold, and someone removes it for just one second at a time.
Full Observability (easy):
Agent sees EVERYTHING about the environment at once
→ Easy to make decisions
Partial Observability (hard, realistic):
Agent only sees a LIMITED snapshot each step
→ Must REMEMBER and INFER what it can't see
Standard VLM strength:
"Here is ONE image. Answer this question."
→ All needed info is RIGHT THERE ✓
Visual Agent challenge:
Step 1: See image A (partial info)
Step 2: See image B (different partial info)
Step 3: See image C (yet more partial info)
→ Must COMBINE all steps to understand true state ✗
The model was never trained to do this kind of cross-step integration.
Partially Observable Markov Decision Process
┌─────────────────────────────────────────────────────┐
│ POMDP Components │
├──────────────────┬──────────────────────────────────┤
│ True State (S) │ The REAL situation in the world │
│ │ (agent CANNOT directly see this) │
├──────────────────┼──────────────────────────────────┤
│ Observation (O) │ What the agent ACTUALLY sees │
│ │ (image + text prompt) │
├──────────────────┼──────────────────────────────────┤
│ Action (A) │ What the agent DOES │
├──────────────────┼──────────────────────────────────┤
│ Transition (T) │ How actions CHANGE the state │
├──────────────────┼──────────────────────────────────┤
│ Reward (R) │ Feedback on how well agent did │
└──────────────────┴──────────────────────────────────┘
True State: Player is at (2,3), hole at (2,4), goal at (4,4)
Observation: Agent sees a partial grid image
Action: Move right
New State: Player is now at (3,3)
Reward: 0 (didn't fall in hole, didn't reach goal yet)
The agent must infer the true state from observations — it can't see it directly.
A world model is an internal mental representation that answers two questions:
Question 1: "Where am I / What is happening RIGHT NOW?"
→ State Estimation
Question 2: "If I do X, what will happen NEXT?"
→ Transition Modeling
When you play chess:
You don't just react — you build a mental model of the game.
Without World Model:
See image → Guess action → Hope for the best
With World Model:
See image → "Current state is X" →
"If I do A, state becomes Y" →
Choose best action deliberately
State estimation means translating a visual observation into a structured description of the current situation.
Visual Input: Structured State Output:
[Grid image] → "Player at (2,3), wall to north,
hole at (2,4), goal at (4,4)"
[Room photo] → "I'm in a bedroom, facing the door,
target couch is not visible"
[Robot camera] → "Gripper is 10cm above red cube,
cube is on the table surface"
The term "grounding" means anchoring abstract concepts to concrete reality. The agent grounds its visual perception into a symbolic, reasoned state description.
Transition modeling means predicting how an action will change the state.
Current State + Action → Predicted Next State
"Player at (2,3)" + "Move Right" → "Player will be at (3,3)"
"In bedroom" + "Go through door" → "Will enter living room"
"Gripper above cube" + "Close gripper" → "Will be holding cube"
Without transition modeling:
Agent takes actions blindly
With transition modeling:
Agent can mentally simulate consequences BEFORE acting
→ Better planning
→ Fewer mistakes
VAGEN structures the agent's output in a specific format:
<think>
<observation>
[State Estimation: What is the current state?]
"Player is at position (2,3). There's a hole at (2,4).
The goal is at (4,4). Path to goal requires going right."
</observation>
[General reasoning about strategy...]
"I should move right to get closer to the goal."
<prediction>
[Transition Modeling: What will happen after my action?]
"If I move right, I'll be at (3,3), which is safe
and one step closer to the goal."
</prediction>
</think>
<answer>move_right</answer>
Purpose 1 (Training):
The structured tags let us EXTRACT and EVALUATE
the reasoning separately from the action
→ We can reward good reasoning directly
Purpose 2 (Execution):
Forces the model to think systematically
→ Better action quality
Less Structured ←————————————————→ More Structured
NoThink FreeThink StateEst. TransModel. WorldModel
| | | | |
Direct Any Only Only Both Est.
action reasoning estimate predict + Predict
1. NoThink:
Input: [image]
Output: <answer>move_right</answer>
Problem: No reasoning at all
2. FreeThink:
Input: [image]
Output: <think>I think I should go right...</think>
<answer>move_right</answer>
Problem: Unstructured, hard to evaluate
3. StateEstimation only:
Output: <think><observation>at (2,3)...</observation></think>
<answer>move_right</answer>
Problem: Knows where it is, but doesn't plan ahead
4. TransitionModeling only:
Output: <think><prediction>moving right → (3,3)</prediction></think>
<answer>move_right</answer>
Problem: Plans ahead, but may misunderstand current state
5. WorldModeling (VAGEN):
Output: <think><observation>...</observation>
<prediction>...</prediction></think>
<answer>move_right</answer>
Best: Understands present AND plans future
┌─────────────────────────────────┐
↓ |
Agent takes action Agent updates
↓ based on reward
Environment responds ↑
↓ |
Agent receives reward ─────────────────┘
Supervised Learning (traditional):
"Here is the correct answer" → Model learns to copy
Problem: We don't always know the "correct" reasoning
Reinforcement Learning:
"Here is how well you did" → Model learns to improve
Advantage: Model discovers its OWN good strategies
Phase 1 - Rollout:
Current model plays through many episodes
Collect all (observation, reasoning, action, reward) data
Phase 2 - Update:
Use collected data to improve the model
Repeat with improved model
This is "on-policy" because we always train on data from
the CURRENT version of the model
Sparse Reward (bad for learning):
Turn 1: action → reward: 0
Turn 2: action → reward: 0
Turn 3: action → reward: 0
...
Turn 20: action → reward: 1 (finally succeeded!)
Problem: How does the model know WHICH of the 20 turns
led to success? Very hard to learn from.
Dense Reward (VAGEN's approach):
Turn 1: action + world model reasoning → reward: 0.8
(state estimation was accurate!)
Turn 2: action + world model reasoning → reward: 0.3
(prediction was wrong)
Turn 3: action + world model reasoning → reward: 0.9
(both estimation and prediction correct!)
Benefit: Feedback at EVERY step, not just at the end
Step 1: Extract agent's state estimation from <observation> tags
Step 2: Extract agent's transition prediction from <prediction> tags
Step 3: Compare against GROUND TRUTH from simulator
Step 4: LLM judge scores the match
Score = Task Success Reward + World Modeling Accuracy Reward
An LLM (language model) acts as an evaluator:
Agent said: "Player is at position (2,3)"
Ground truth: "Player is at position (2,3)"
LLM Judge: "Match! Score: 1.0"
Agent said: "Player is at position (1,1)"
Ground truth: "Player is at position (2,3)"
LLM Judge: "No match. Score: 0.0"
When something goes right (or wrong), which specific decisions deserve the credit (or blame)?
Multi-turn episode:
Turn 1: Moved north ← Did this help?
Turn 2: Moved east ← Did this help?
Turn 3: Moved south ← Did this help?
Turn 4: Fell in hole ← FAILURE
Which turn caused the failure?
Turn 3 was the mistake, but how does the model know?
Within each turn, the model generates many tokens:
Turn 2 output (many tokens):
"The player is at... (2,3)... there is a wall...
to the north... I should... move... east...
because... the goal... is... to the right..."
<answer>move_east</answer>
Which tokens were important?
The reasoning? The action? Both?
Advantage measures: "How much better was this action than average?"
Value function V(s): "How good is this situation on average?"
Q(s,a): "How good is taking action a in situation s?"
Advantage A(s,a) = Q(s,a) - V(s)
= "Was this action BETTER or WORSE than expected?"
Positive advantage → action was better than expected → reinforce it
Negative advantage → action was worse than expected → discourage it
GAE is a technique to estimate advantages while balancing two problems:
Problem 1 - Bias: Using only immediate reward ignores future consequences
Problem 2 - Variance: Using all future rewards is noisy and unstable
GAE solution: Weighted combination that balances both
(controlled by parameter λ)
Level 1: TURN-LEVEL
"Which turns in the episode were most important?"
Episode: Turn1 → Turn2 → Turn3 → Turn4 → Success
Credits: 0.1 0.2 0.8 0.9 (Turn 3&4 mattered most)
Level 2: TOKEN-LEVEL
"Within each turn, which tokens mattered most?"
Turn 3: "state is X... [reasoning tokens]... move right [action token]"
Credits: 0.3 0.3 0.9
(action token mattered most within this turn)
Single-level GAE (standard):
Treats all tokens equally across the entire episode
→ Loses the structure of multi-turn reasoning
→ Imprecise credit assignment
Bi-Level GAE (VAGEN):
First: Identify important TURNS
Then: Within important turns, identify important TOKENS
→ More precise
→ Better learning signal
Step 1: Compute turn-level advantages
(using rewards at turn boundaries)
Turn 1 advantage: low (early, uncertain)
Turn 2 advantage: low (still early)
Turn 3 advantage: high (pivotal decision!)
Step 2: For each turn, compute token-level advantages
(using the turn-level advantage as context)
Turn 3, token "move": high advantage
Turn 3, token "because": lower advantage
Turn 3, token "the": very low advantage
Step 3: Use these precise advantages to update the model
To prove the approach works generally, not just on one specific problem.
Task 1: FrozenLake (Grid Navigation)
┌─┬─┬─┬─┐
│S│ │ │ │ S = Start, G = Goal
│ │H│ │H│ H = Hole (avoid!)
│ │ │ │H│
│H│ │ │G│
└─┴─┴─┴─┘
Tests: Position tracking, path planning
Task 2: Sokoban (Puzzle Solving)
Push boxes onto targets
Tests: Multi-step planning, spatial reasoning
Task 3: Visual Navigation (3D Environments)
Find objects in realistic indoor scenes
Tests: Visual understanding, exploration
Task 4: ManiSkill (Robot Manipulation)
Control robot arm to pick/place objects
Tests: 3D spatial reasoning, precise control
Task 5: SVG Construction (Visual-to-Code)
Generate code that recreates a target image
Tests: Visual understanding, iterative refinement
Model Size vs. Performance:
Qwen2.5-VL-72B (72 billion params): 0.51 overall
GPT-5 (unknown, very large): 0.75 overall
VAGEN (3 billion params): 0.82 overall
Insight: A 3B model with good TRAINING beats a 72B model
and even the best proprietary models!
Strategy Overall Score Improvement
─────────────────────────────────────────────
NoThink ~0.28 baseline
FreeThink ~0.67 +139%
StateEstimation ~0.56 +100%
TransitionModel ~0.68 +143%
WorldModeling ~0.76 +171%
VAGEN-Full ~0.82 +193%
WorldModeling Reward alone:
✓ Helps consistently
✗ Limited by poor credit assignment
Bi-Level GAE alone:
✓ Notable improvements
✗ Unstable without dense rewards
Both together (VAGEN-Full):
✓ Strong AND stable
✓ Best across all tasks
Lesson: The two components are COMPLEMENTARY
┌─────────────────────────────────────────────────────────────┐
│ VAGEN Complete Picture │
├─────────────────────────────────────────────────────────────┤
│ │
│ PROBLEM: VLMs fail as multi-turn agents because of │
│ partial observability │
│ │
│ SOLUTION: Teach agents to build WORLD MODELS │
│ │
│ HOW: │
│ 1. Structured reasoning format │
│ (observation + prediction tags) │
│ │
│ 2. WorldModeling Reward │
│ (dense feedback via LLM judge) │
│ │
│ 3. Bi-Level GAE │
│ (precise credit at turn AND token level) │
│ │
│ RESULT: 3B model beats GPT-5, Gemini 2.5 Pro, Claude 4.5 │
│ │
│ KEY INSIGHT: Better TRAINING > Bigger MODEL SIZE │
└─────────────────────────────────────────────────────────────┘
| # | Concept | One-Line Summary |
|---|---|---|
| 1 | VLMs | AI that understands both images and text |
| 2 | Visual Agents | VLMs that take actions over multiple steps |
| 3 | Partial Observability | Agent only sees limited snapshots, not full state |
| 4 | POMDP | Formal framework for partial observability problems |
| 5 | World Models | Internal representation of current state + future transitions |
| 6 | State Estimation | "What is happening right now?" |
| 7 | Transition Modeling | "What will happen if I do X?" |
| 8 | Structured Reasoning | XML tags force systematic thinking |
| 9 | Five Strategies | Spectrum from no reasoning to full world modeling |
| 10 | Reinforcement Learning | Learn from rewards, not labeled examples |
| 11 | WorldModeling Reward | Dense per-turn feedback on reasoning accuracy |
| 12 | Credit Assignment | Which decisions caused success/failure? |
| 13 | GAE | Estimate action quality while balancing bias/variance |
| 14 | Bi-Level GAE | Hierarchical credit at turn-level then token-level |
| 15 | Five Tasks | Diverse benchmarks proving general applicability |
| 16 | Results | 3B VAGEN model beats all proprietary models |