After studying this material, you should be able to:
- Explain the cold-start problem in recommendation systems and why it matters
- Describe how multimodal embeddings solve cold-start in personalization
- Understand how CLIP enables cross-modal and cross-canvas signal transfer
- Explain reward-based weighting and why it matters for mixed training data
- Understand inverse propensity scoring (IPS) as an unbiased offline evaluation method
- Describe how multimodal foundation models (MediaFM) improve video personalization
- Explain the linear probe as a cheap embedding screening tool
- Understand the role of shared embedding infrastructure in scalable ML systems
Step-by-Step Study Material
Step 1: The Core Problem — What Is Cold-Start?
The Situation
Imagine Netflix needs to show you one artwork image for a new show. They have thousands of members and dozens of possible artwork options. The goal is to show you specifically the artwork most likely to make you click and watch.
How Traditional Systems Work
Traditional recommendation models learn from interaction history:
Member sees Artwork A → clicks → model learns "Artwork A is good"
Member sees Artwork B → ignores → model learns "Artwork B is bad"
The model treats each artwork as an opaque ID — just a number with no meaning attached to it.
The Cold-Start Problem
New Title Launches
↓
New Artworks Created (Artwork X, Y, Z)
↓
Model sees: ID_X, ID_Y, ID_Z
↓
Model thinks: "I have NEVER seen these IDs before"
↓
Model cannot personalize → falls back to popularity heuristics
The problem: The model is blind to what is actually in the artwork. It cannot use your known preferences because it has never seen these specific IDs before.
Why This Matters
- Every new title launch suffers degraded personalization
- Members see generic, non-personalized artwork
- Discovery quality drops exactly when it matters most — at launch
Key Insight: The cold-start problem is not about lacking member data. It is about the model being unable to connect what it knows about your taste to a brand-new asset it has never seen before.
Step 2: The Solution — Multimodal Embeddings
What Is an Embedding?
An embedding is a dense numerical vector that represents something in a way that captures its meaning.
"A photo of a comedian on stage"
↓ [embedding model]
↓
[0.23, -0.87, 0.45, 0.12, ... ] ← 768 numbers
Similar things produce similar vectors. This is the key property.
What Is CLIP?
CLIP (Contrastive Language-Image Pretraining) is a model trained to understand both images and text in a shared embedding space.
Image of comedian → CLIP → vector_A
Text "comedian performing" → CLIP → vector_B
vector_A ≈ vector_B (they are close in embedding space)
Critical property: CLIP was pretrained on massive data, so it already understands visual concepts before Netflix trains anything.
How Netflix Uses CLIP for Artwork
Instead of representing an artwork only by its ID, they combine two things:
Asset Representation = [CLIP image embedding] + [learned ID embedding]
(768-dim vector) (learned from data)
"what it looks like" "what interactions say"
What this achieves:
| Situation | Old Model | New Model |
|---|---|---|
| New artwork, no history | Blind — random guess | Uses CLIP embedding immediately |
| Member likes comedian X | Only works for seen IDs | Transfers to any artwork featuring X |
| Similar artworks | Treated as unrelated | Recognized as similar via embedding |
The Transfer Learning Effect
This is the most important concept in this section:
You consistently click artworks featuring Comedian X
↓
Model learns: "This member likes embeddings that look like [comedian X vector]"
↓
New title launches featuring Comedian X
↓
New artwork has CLIP embedding similar to [comedian X vector]
↓
Model immediately knows: "Show this member this artwork"
The preference is stored in embedding space, not tied to specific IDs. Therefore it transfers across titles automatically.
Step 3: From Five Models to One — Model Consolidation
The Original Problem
Netflix artworks come in multiple canvas types (shapes/sizes):
- Billboard
- Vertical-box
- Horizontal-panel
- Short-panel
- Landscape-panel
Previously: One separate model per canvas
Why Separate Models Were Necessary Before
With ID-based models, the model had no way to know that:
Artwork_A_billboard (ID: 10045)
Artwork_A_vertical (ID: 10046) ← Same scene, different crop
Artwork_A_landscape (ID: 10047)
...are all the same image cropped differently. They were three completely unrelated IDs.
How CLIP Solves This
CLIP embeddings are invariant to crop, resize, and aspect ratio:
Same scene, billboard crop → CLIP → [0.23, -0.87, 0.45, ...]
Same scene, vertical crop → CLIP → [0.23, -0.88, 0.44, ...] ← Nearly identical
Same scene, landscape crop → CLIP → [0.24, -0.87, 0.45, ...]
Because the vectors are nearly the same, one unified model can now:
- Learn from interactions on high-traffic canvases (billboard)
- Apply that learning to low-traffic canvases (short-panel)
- Treat all crops of the same scene as related
Result: 5 models → 1 unified model
Why This Especially Helps Sparse Canvases
Billboard canvas: 1,000,000 impressions/day → lots of learning signal
Short-panel canvas: 10,000 impressions/day → very little signal
Old system: Short-panel model starved of data
New system: Short-panel benefits from billboard's 1,000,000 impressions
Step 4: Reward-Based Weighting — Mixing Training Data Fairly
The New Problem Created by Consolidation
When you pool data from all canvases, you face an imbalance:
Billboard: 1,000,000 examples
Short-panel: 10,000 examples
If you train naively, the model mostly learns from billboard data. Short-panel barely influences training — the opposite of what you want.
Naive Solutions and Why They Fail
| Approach | Problem |
|---|---|
| Train on raw counts | High-volume canvas dominates |
| Hand-tune weights per canvas | Arbitrary, requires constant tuning |
| Undersample high-volume | Wastes valuable data |
The Reward-Based Weighting Solution
Instead of weighting by how many interactions occurred, weight by how valuable each interaction is:
Weight of training example = long-term reward score of that interaction type
Intuition:
Canvas A: 1,000,000 low-value clicks (weight = 0.1 each)
Canvas B: 10,000 high-value plays (weight = 1.0 each)
Effective contribution:
Canvas A: 1,000,000 × 0.1 = 100,000
Canvas B: 10,000 × 1.0 = 10,000
The rebalancing happens automatically based on value, not volume. No manual tuning required.
Key benefit: The model optimizes for long-term member satisfaction, not for whichever action happens most frequently.
Step 5: Offline Evaluation — Inverse Propensity Scoring (IPS)
The Evaluation Problem
You have trained a new model. Before running an expensive A/B test, you want to evaluate it offline using historical data. But there is a fundamental bias problem:
Production policy shows Artwork A 90% of the time
Production policy shows Artwork B 10% of the time
Historical data: mostly interactions with Artwork A
New model prefers Artwork B
Evaluation result: New model looks bad
Reality: New model might be better, but we barely showed Artwork B
The logged data describes what the current policy preferred, not what members would have chosen from all options.
The Solution: Exploration Traffic + IPS
Step 1: Create exploration traffic
A small fraction of traffic uses a randomized policy that shows artworks with known, logged probabilities:
Artwork A shown with probability 0.33 (logged exactly)
Artwork B shown with probability 0.33 (logged exactly)
Artwork C shown with probability 0.33 (logged exactly)
Step 2: Inverse Propensity Scoring
For each observation, upweight rare impressions and downweight common ones:
IPS estimate = Σ [reward(x,a) / propensity(a|x)]
Example:
Artwork B was shown with propensity 0.1 (rarely shown)
Member clicked → reward = 1
IPS contribution = 1 / 0.1 = 10 (upweighted because it was rare)
Artwork A was shown with propensity 0.9 (frequently shown)
Member clicked → reward = 1
IPS contribution = 1 / 0.9 = 1.1 (barely upweighted)
Why this works: Rare impressions are upweighted to compensate for being underrepresented. The result is an unbiased estimate of how a new policy would perform if actually deployed.
Why Known Propensities Matter
Netflix logs propensities at serving time from the randomized policy. They do not estimate them afterward. This is critical:
Known propensity (logged): exact, unbiased
Estimated propensity (modeled after the fact): approximate, introduces error
This is described as "the single biggest reason our offline numbers track online outcomes."
Step 6: Ablation Study — Why Both Ideas Are Needed Together
The Three Variants Tested
| Version | Description |
|---|---|
| V1 | CLIP embeddings + separate per-canvas models |
| V2 | No CLIP embeddings + unified single model |
| V3 | CLIP embeddings + unified single model (both ideas) |
Results Pattern
V1 alone: Small improvement, not statistically significant online
V2 alone: Small improvement, not statistically significant online
V3 combined: Statistically significant lift online
Why V1 alone fails:
CLIP tells the model what an artwork looks like
BUT
Sparse canvas has too few examples to learn how to USE that information
Why V2 alone fails:
Unified model has plenty of data from all canvases
BUT
Data is ID-based, so new assets with no ID history still have cold-start
Why V3 succeeds:
Mature canvases teach the model how CLIP embeddings map to member preference
That mapping transfers to sparse canvases immediately
New assets arrive with CLIP embeddings the model already knows how to use
Key Lesson: When a new feature does not help, look for a second blocking factor before concluding the feature is useless. V1 was blocked by data sparsity. V2 was blocked by ID-only representation. V3 removed both blockers simultaneously.
Step 7: Query-Aware Artwork Ranking
The Problem
Your general taste is the right signal when browsing. But when searching, your intent is explicit.
You generally like: action movies
You search for: "Emma Stone"
Best artwork to show: one featuring Emma Stone
(even if your general taste would suggest an action scene)
The Elegant Solution
Because CLIP projects both text and images into the same embedding space:
Query text "Emma Stone" → CLIP text encoder → text_vector
Artwork image → CLIP image encoder → image_vector
Cosine similarity(text_vector, image_vector) = how well artwork matches query
The final ranking score blends two signals:
Score = α × personalization_score + (1-α) × query_artwork_alignment
Where:
- personalization_score = what the artwork model thinks you like
- query_artwork_alignment = cosine similarity between query text and artwork image
- α = mixing weight tuned via A/B testing (between 0 and 1)
Why this is powerful: No new modeling effort was required. The CLIP embeddings already existed from the artwork work. Adding one similarity term at scoring time created a query-aware ranker.
Step 8: Video Preview Personalization
Why Video Is Harder Than Images
A video preview's appeal comes from:
- Visual content (frames)
- Motion and pacing
- Dialogue
- Soundtrack and music
A mean of still-frame CLIP embeddings (SeqCLIP) captures only visual content. It misses everything else.
SeqCLIP (First Attempt)
Video preview → extract frames → CLIP encode each frame → average all vectors
Result: One vector representing "what the video looks like"
Limitation: Ignores audio, dialogue, pacing — major components of a preview's appeal.
MediaFM (Netflix's Foundation Model)
MediaFM processes video shot by shot, fusing three signals per shot:
Each shot → [visual frames] + [audio] + [timed text/dialogue]
↓
MediaFM fusion
↓
Single shot embedding
Multiple shot embeddings are combined into one video preview embedding.
Results Ordering
MediaFM > SeqCLIP > ID-only baseline
Each step of added content awareness helped. Audio and dialogue signals added real value beyond visual-only encoding.
Step 9: The Linear Probe — Cheap Embedding Screening
The Problem
New embeddings arrive constantly. Full evaluation pipeline is expensive:
Full pipeline cost:
Data engineering → Model retraining → Weeks of A/B test traffic
You cannot afford to run this for every candidate embedding.
The Linear Probe Solution
Question: From the embedding alone, can you predict which asset wins under a popularity-based policy?
Process:
Step 1: Select fixed set of titles
Step 2: Use exploration data + IPS to find debiased popularity winner per title
Step 3: Label: winner = 1, others = 0
Step 4: Train a LINEAR classifier on embeddings only
Step 5: Measure accuracy vs. random baseline
Why linear?
A linear probe is intentionally simple. If a complex model is needed to extract signal, the embedding is not encoding that signal cleanly. A good embedding should allow a simple linear classifier to identify winners.
High linear probe accuracy → embedding captures semantic drivers of popularity
Low linear probe accuracy → embedding does not encode useful information
How It Gates the Pipeline
Many candidate embeddings
↓
Linear probe screening (cheap)
↓
2-3 finalists
↓
Full offline IPS evaluation (moderate cost)
↓
1-2 candidates
↓
Online A/B test (expensive)
Validation: For MediaFM vs. SeqCLIP, all three signals (linear probe, IPS, A/B test) agreed on the same ranking. This validates the probe as a reliable early-stage filter.
Step 10: Shared Infrastructure — The Embedding Store
The Architecture
Raw asset (image/video)
↓
Foundation model (CLIP / MediaFM)
↓
Dense embedding vector
↓
Netflix Embedding Store ← single source of truth
↓
┌───────────────────────────────────────┐
│ Artwork model │ Query ranker │ Video preview model │ Others...
└───────────────────────────────────────┘
Key Properties
1. Encode once, serve everywhere The foundation model runs once per asset. Every downstream system reads the same vector.
2. Training-serving consistency The exact same embeddings are used at training time and inference time. No skew between what the model learned from and what it sees in production.
3. Decoupled updates
New embedding version registered
↓
Backfilled across catalog
↓
Validated independently
↓
Available to all models via configuration
(no code changes in downstream models)
This is what allowed Netflix to:
- Swap CLIP into the artwork model
- Stand up the query-aware ranker on the same vectors
- Roll MediaFM into the video preview model
...each as an independent change rather than a coordinated cross-team migration.
Summary: The Complete MAPS System
MAPS: Multimodal Asset Personalization at Scale
Problem: Cold-start — new assets have no interaction history
Solution Architecture:
┌─────────────────────────────────────────────────────┐
│ Embedding Store │
│ CLIP (images) ──────────────────────────────────┐ │
│ MediaFM (video: visual + audio + text) ─────────┤ │
└──────────────────────────────────────────────────┼──┘
│
┌────────────────────────────────────┤
↓ ↓ ↓
Artwork Model Query-Aware Ranker Video Preview Model
(5→1 canvas) (personalization (ID → content-aware)
+ reward weights + query alignment)
↓ ↓ ↓
IPS offline evaluation → A/B test → Production
↑
Linear probe gates new embeddings cheaply
Three Core Lessons
| Lesson | Description |
|---|---|
| Compound effects | Two ideas that each fail alone can succeed together — look for blocking factors |
| Shared embeddings | Encode once, reuse everywhere — infrastructure investment pays across all systems |
| Gate expensively | Use cheap proxies (linear probe) to filter before committing to expensive evaluation |