After studying this material, students should be able to:
Agent → takes Action → Environment → gives Reward + New State
↑___________________________________|
An agent learns a policy (strategy) to maximize cumulative rewards.
Instead of only learning from real experience:
| Approach | How it learns |
|---|---|
| Model-Free RL | Only real environment interactions |
| Model-Based RL | Learns a model of the environment, then simulates/plans with it |
Key Insight: A learned model lets the agent "imagine" outcomes without costly real-world interactions.
To improve a policy, we need to compute gradients — the direction to adjust policy parameters to get more reward.
Policy Parameters (θ) → Actions → Rewards
We need: "How does changing θ affect total reward?"
There are two fundamentally different approaches:
Also called REINFORCE or "derivative-free"
Core Idea: Don't differentiate through the environment. Instead, observe outcomes and weight them by probability.
Intuition:
- Try many actions
- Actions that led to HIGH reward → increase their probability
- Actions that led to LOW reward → decrease their probability
Formula concept:
∇θ J ≈ E[∇θ log π(a|s) · R]
↑ ↑
"How likely was "How good was
this action?" the outcome?"
Pros and Cons:
| ✅ Pros | ❌ Cons |
|---|---|
| Works even if model is non-differentiable | Very HIGH variance |
| Simple to implement | Needs many samples to be accurate |
| General purpose | Slow convergence |
Also called "pathwise gradients" or backpropagation through time
Core Idea: If the model is differentiable, backpropagate gradients directly through the simulation.
θ → action → model → next state → model → ... → Reward
←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←←
Backpropagate gradients directly
Intuition: Like training a neural network — chain rule all the way back.
Pros and Cons:
| ✅ Pros | ❌ Cons |
|---|---|
| LOW variance | Requires differentiable model |
| Efficient use of data | Gradients can explode or vanish over long horizons |
| Fast convergence | Numerically unstable for long sequences |
LR Estimator: [============================] High Variance, works anywhere
RP Estimator: [====] Low Variance, but unstable long-term
Combined: [========] Best of both worlds?
Core Idea: At each step in a trajectory, choose or blend between LR and RP based on which is more reliable.
Trajectory: s₀ → s₁ → s₂ → s₃ → s₄ → ... → sₙ
Use RP: [←←←←←←←←←] (recent steps, stable)
Use LR: [←←←←←←←←←←←←←←←←←←←←←] (earlier steps, RP unstable)
Key Insight: RP gradients are reliable for short horizons but degrade over long ones. LR can cover the rest.
IVW is a statistical technique to optimally combine multiple estimators of the same quantity.
Intuition: Trust estimators that are more consistent (low variance) more heavily.
If estimator 1 has variance σ₁²
If estimator 2 has variance σ₂²
Combined weight for estimator 1: w₁ = (1/σ₁²) / (1/σ₁² + 1/σ₂²)
Combined weight for estimator 2: w₂ = (1/σ₂²) / (1/σ₁² + 1/σ₂²)
Final estimate = w₁·estimate₁ + w₂·estimate₂
Estimator A: variance = 1 → weight = 1/1 = 1.0 → 50% weight
Estimator B: variance = 0.1 → weight = 1/0.1 = 10 → ~91% weight
→ Trust B much more because it's more consistent!
TP applies IVW at every node along the trajectory to blend LR and RP gradients.
Trajectory nodes:
s₀ ——→ s₁ ——→ s₂ ——→ s₃ ——→ Reward
↑ ↑ ↑ ↑
IVW IVW IVW IVW
(blend LR+RP at each point)
Orders of magnitude improvement over using either LR or RP alone
Improvement 1: Different Node for IVW
TP: Apply IVW at every intermediate node
TPX: Apply IVW at a strategically chosen node
→ Less computation, same benefit
Improvement 2: Coordinate-wise Weighting
TP: Single weight for entire gradient vector
[w · (g₁, g₂, g₃, g₄, g₅)]
TPX: Different weight per dimension
[(w₁·g₁, w₂·g₂, w₃·g₃, w₄·g₄, w₅·g₅)]
Why coordinate-wise matters:
| Feature | TP | TPX |
|---|---|---|
| IVW application | All nodes | Strategic node |
| Weighting | Scalar | Per-coordinate |
| Scalability | Limited | ✅ Modern tasks |
| Implementation | Complex | ✅ Easier |
Dreamer is a state-of-the-art visual MBRL algorithm:
Real pixels → Encoder → Latent State → World Model → Imagined Future
↓
Policy Gradient
Problem with standard Dreamer:
Short horizon: ✅ Works well
Long horizon: ❌ Fails (RP gradients explode/vanish)
With TPX:
Short horizon: ✅ Works well
Long horizon: ✅ Works reliably
Cost: Only a fraction of additional computation
MBRL Problem
│
▼
Need Policy Gradients
│
├──→ LR Estimator (high variance, general)
│
├──→ RP Estimator (low variance, unstable long-term)
│
└──→ COMBINE THEM (best of both)
│
▼
IVW Weighting
(trust low-variance more)
│
▼
TP (prior work, not scalable)
│
▼
TPX (scalable, coordinate-wise)
│
▼
Applied to Dreamer
✅ Solves long-horizon failure
✅ Minimal extra cost
✅ Easy to implement
| Concept | Remember This |
|---|---|
| LR gradients | High variance, works anywhere, derivative-free |
| RP gradients | Low variance, needs differentiable model, unstable long-term |
| IVW | Combine estimators by trusting lower-variance ones more |
| TP | First composite method, proved concept, not scalable |
| TPX | Scalable TP with coordinate-wise weighting |
| Key result | Long horizons that break Dreamer work with TPX |