Step-by-Step Study Material

Step 1: Foundation — What is RLHF?

The Core Problem

Training AI agents requires a reward signal, but defining reward functions manually is hard.

Example: How do you numerically define "write a helpful response"?

The RLHF Solution

Instead of defining reward manually:

1. Show a human two different behaviors (trajectory segments)
2. Human picks which one they prefer
3. Algorithm learns a reward function from these preferences
4. Agent trains using that learned reward

Trajectory A: [s1→s2→s3]  ←── Human picks one
Trajectory B: [s1→s4→s5]  ←── 
         ↓
    Reward Function Learned
         ↓
    Agent Trained

---

Step 2: Key Vocabulary

---

Step 3: The Two Preference Models

Model 1: Partial Return Model (Traditional Assumption)

Assumption: Humans prefer whichever segment gave them more reward

$P(A \succ B) = \sigma\left(\sum_{t \in A} r(s_t, a_t) - \sum_{t \in B} r(s_t, a_t)\right)$

Where σ is the sigmoid function

Intuition:

Segment A earned 10 points → Human prefers A
Segment B earned  3 points → Human rejects B

Problem: This assumes humans only care about what happened, not what could have happened

---

Model 2: Regret-Based Model (Alternative)

Assumption: Humans prefer whichever segment had less regret

$\text{Regret}(segment) = \sum_t [r^*(s_t) - r(s_t, a_t)]$

Where r*(s) = reward from taking the optimal action in state s

Intuition:

Segment A: Got 10 points, but COULD have gotten 10 → Low regret ✓
Segment B: Got 8 points, but COULD have gotten 8  → Low regret ✓
Segment C: Got 10 points, but COULD have gotten 20 → HIGH regret ✗

Key Insight: Humans naturally think "that was a mistake" — comparing actions to what should have been done

---

Step 4: The Central Problem — Model Mismatch

What Happens When You're Wrong About the Model?

REALITY:          Humans use REGRET model
ASSUMPTION:       Algorithm assumes PARTIAL RETURN model
CONSEQUENCE:      ???

This paper answers that question mathematically.

The Paper's Core Argument

When you:

• Assume partial return model
• But humans actually use regret model
• Then the function you learn ≈ Optimal Advantage Function

$\hat{r}(s,a) \approx A^*(s,a)$

Not the true reward r(s,a)!

---

Step 5: Understanding the Optimal Advantage Function

Advantage Function Definition

$A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$

Optimal Advantage

$A^(s,a) = Q^(s,a) - V^*(s)$

Uses the optimal policy as the baseline

A*(s,a) > 0  →  Action a is BETTER than optimal baseline
A*(s,a) = 0  →  Action a IS the optimal action
A*(s,a) < 0  →  Action a is WORSE than optimal baseline

---

Step 6: Why Does This Happen? The Intuition

Connecting Regret to Advantage

Regret of an action = how far below optimal you are $\text{Regret}(s,a) = V^(s) - Q^(s,a) = -A^*(s,a)$

Therefore:

• Minimizing regret = Maximizing optimal advantage
• Preferences based on regret → learned function captures optimal advantage

Visual Analogy

TRUE REWARD landscape:        OPTIMAL ADVANTAGE landscape:
    
    🏔️  (absolute height)         📐 (height relative to best path)
    
Partial Return measures:      Regret measures:
"How high did you climb?"     "How far below the peak were you?"

---

Step 7: Why This Matters — Practical Consequences

Scenario Analysis

The Silver Lining

If your goal is finding the optimal policy (not the true reward), learning A* might still work!

Because:

• Optimal actions have A*(s,a) = 0 (highest possible)
• Suboptimal actions have A*(s,a) < 0
• Ranking is preserved → policy optimization still works

The Danger

If you need the true reward (for safety, interpretability, reward transfer), you have a problem

Learned: "This action scores +5"  (advantage)
Reality: "This action scores +105" (true reward, baseline=100)

---

Step 8: Summary Framework

┌─────────────────────────────────────────────────────┐
│                    RLHF Pipeline                     │
├─────────────────────────────────────────────────────┤
│                                                      │
│  Human Preferences                                   │
│       │                                              │
│       ▼                                              │
│  [Partial Return Model Assumed]                      │
│       │                                              │
│       ▼                                              │
│  Learned Function r̂(s,a)                            │
│       │                                              │
│  IF humans used REGRET:  r̂ ≈ A*(s,a)  ← THIS PAPER │
│  IF humans used RETURN:  r̂ ≈ r(s,a)   ← Traditional│
│       │                                              │
│       ▼                                              │
│  Policy Optimization                                 │
│  (May still work, but reward is misidentified)       │
└─────────────────────────────────────────────────────┘

---

Step 9: Key Takeaways

Remember These Three Points:

1. Model Assumption Matters

   The preference model you assume determines what function you actually learn

2. Regret ≠ Return

   Humans naturally compare to optimal behavior, not just accumulate rewards

3. Optimal Advantage ≠ Reward

   You can optimize policy correctly but still misidentify the reward function

---

Self-Check Questions

1. What is the difference between partial return and regret as bases for human preference?

2. Why might humans naturally use a regret-based model when comparing trajectories?

3. If you learn A*(s,a) instead of r(s,a), under what conditions does your policy optimization still succeed?

4. Give an example where mistaking A* for r would cause a real problem in an AI system.

5. What does it mean mathematically that *Regret(s,a) = -A(s,a)**?