After studying this material, you should be able to:

1. Explain why measuring the causal impact of recommendations is difficult
2. Describe the hybrid modeling approach used to estimate recommendation value
3. Understand how the low-rank choice model works and why it is necessary
4. Interpret counterfactual analysis results comparing recommendation systems
5. Decompose recommendation-driven viewing into its three distinct channels
6. Evaluate trade-offs between engagement and consumption diversity

---

Step-by-Step Study Material

Step 1: The Core Problem — Why Is Measurement Hard?

The Naive Approach and Why It Fails

A natural first instinct is to ask:

"What share of viewing comes from recommended titles?"

This seems reasonable, but it is fundamentally flawed because it confuses correlation with causation.

The Targeting Problem

Member watches a recommended title
         ↓
Was it because of the recommendation?
         OR
Would she have watched it anyway?

Recommendations are not randomly assigned. The system deliberately shows titles it predicts a member will enjoy. Therefore:

Observed viewing = Underlying preference + Recommendation effect

You cannot separate these two components just by looking at viewing data alone.

Why You Cannot Simply Turn Off Personalization

The obvious experimental solution — replace personalized recommendations with random ones at scale — is rejected because:

• Personalization is central to member experience
• It would harm millions of users during the experiment
• It is not an ethically or commercially acceptable strategy

Key Takeaway from Step 1

The measurement challenge is that recommendations are targeted toward demand, making it impossible to isolate their causal effect through simple observation.

---

Step 2: The Hybrid Solution — Model + Experiment

Since you cannot run a full-scale experiment, the researchers use a two-part hybrid approach:

Part A: Build a Behavioral Model

The model uses natural variation in the recommendation system — for example, when the algorithm explores new options — to estimate how recommendations affect choices.

Part B: Validate with Small-Scale A/B Tests

Rather than turning off personalization entirely, researchers run small nudge experiments:

• Slightly increase or decrease the visibility of specific title categories
• Measure how viewing shifts as a result
• Compare these real experimental results to what the model predicted

This answers the critical question:

Does the model actually capture how members substitute between titles?

Why This Hybrid Approach Works

Large-scale model  →  Scale and richness
        +
Small A/B tests    →  Credibility and ground truth
        =
Validated counterfactual engine

Key Takeaway from Step 2

The hybrid approach combines the breadth of modeling with the credibility of experiments, allowing causal inference without destroying the member experience.

---

Step 3: Understanding the Low-Rank Choice Model

What the Model Needs to Capture

A simple model would assume that removing one title shifts viewing proportionally to all other titles. But this is unrealistic:

• A member considering a romantic comedy will substitute toward other romantic comedies, not action films
• Two members may react very differently to the same recommendation

The Low-Rank Choice Model

Instead of using manually defined features like genre or cast, the model learns representations directly from viewing behavior:

Each title  →  Learned vector representation
Each member →  Learned preference vector
         ↓
Similarity between member and title
predicts viewing probability

This captures nuanced similarities that coarse labels like genre cannot describe.

Dynamic Preferences

Member preferences are not fixed. What someone wants to watch depends on recent viewing history:

Finished a crime thriller  →  Preference state A
Finished a stand-up special →  Preference state B
Finished a kids' movie     →  Preference state C

The model summarizes viewing history into a current preference state, allowing it to scale to millions of members while capturing changing tastes.

The Recommendation Bonus

When a title is recommended, the model adds an explicit recommendation effect (a "bonus") to its selection probability. This bonus captures two mechanisms:

Key Takeaway from Step 3

The low-rank choice model learns flexible, data-driven representations of members and titles, and explicitly separates the recommendation effect from underlying preferences.

---

Step 4: Validating the Model

The Critical Assumption

The model assumes that, after accounting for all information the recommender system uses, the remaining variation in recommendations is effectively random. This is plausible because the researchers observe all inputs to the recommender system.

However, an assumption alone is not enough. The model must be tested.

The Validation Experiment

Researchers ran experiments that:

1. Randomly assigned members to treatment groups
2. Slightly nudged recommendations toward or away from certain title categories
3. Measured diversion ratios — when titles become less visible, where does viewing go?

Title becomes less visible
         ↓
Does viewing shift to similar titles?
         OR
Does the member stop watching altogether?

Validation Results

This strong agreement confirms that the model captures real substitution patterns, not just historical patterns.

Key Takeaway from Step 4

Validation against randomized experiments gives credibility to the model's counterfactual predictions, which is essential before drawing conclusions about different recommendation systems.

---

Step 5: Counterfactual Analysis — How Much Does Personalization Matter?

The Three Baselines

The current Netflix recommender system is compared against three alternatives:

Engagement Results

Even the comparison to matrix factorization — itself a personalized system — shows that continued improvements in personalization produce meaningful gains.

Consumption Diversity Results

Engagement alone is not sufficient. A system could increase viewing by concentrating attention on a few popular titles, which would poorly serve members with niche tastes.

High engagement + Low diversity  =  Bad for catalog breadth
High engagement + High diversity =  Good for members and catalog

Key Takeaway from Step 5

The current recommender system achieves both higher engagement and broader consumption diversity, meaning it serves a wider range of member tastes simultaneously.

---

Step 6: Decomposing Why Recommended Titles Get Watched

The Three Channels

Not all recommendation-driven viewing has the same cause. The model decomposes viewing into three distinct channels:

Visualizing the Decomposition

Total recommendation-driven viewing
              ↓
    ┌─────────────────────┐
    │  Exposure (7%)      │  ← Mechanical visibility
    │  Selection          │  ← Finding existing demand
    │  Targeting          │  ← Creating new demand
    └─────────────────────┘
         Targeting ≈ 7× Exposure

How This Changes Across Systems

Where Targeting Is Strongest

Targeting is not uniform across the catalog:

Very popular titles   →  Low targeting value (everyone knows them)
Very niche titles     →  Low targeting value (audience too small)
Mid-popularity titles →  HIGHEST targeting value

Mid-popularity titles have enough potential audience to be broadly relevant, but need personalization to connect with the right members.

Key Takeaway from Step 6

The primary value of advanced personalization is targeting — placing the right title in front of the right member at the right moment — not simply making titles visible.

---

Summary: Connecting All Steps

Step 1: Identify the problem
        Recommendations are targeted, so simple metrics are biased

Step 2: Design the solution
        Hybrid approach = behavioral model + small A/B tests

Step 3: Build the model
        Low-rank choice model with dynamic preferences and recommendation bonus

Step 4: Validate the model
        Experimental diversion ratios confirm model accuracy (r = 0.86)

Step 5: Run counterfactuals
        Current system outperforms all baselines on engagement AND diversity

Step 6: Decompose the mechanism
        Value comes primarily from targeting, not exposure

---

Concept Check Questions

Test your understanding with these questions:

1. Why does measuring "share of viewing from recommended titles" fail to answer the causal question?
2. What is a diversion ratio, and why is it useful for validation?
3. Why does the low-rank model learn representations from behavior rather than using predefined features like genre?
4. What is the difference between selection and targeting as channels of recommendation value?
5. Why do mid-popularity titles benefit most from targeting?
6. What would it mean if a recommender system increased engagement but reduced diversity?