Step-by-Step Study Material
Step 1: Understand the Problem Context
What is Ad Event Enrichment?
Think of it like a two-piece puzzle:
Ad Server Side Client Device Side
───────────────── ──────────────────
Knows EVERYTHING: Knows almost NOTHING:
• Campaign details • Opaque token
• Creative info • When to send events
• Price • "Ad was watched"
• Tracking URLs
Enrichment = reuniting these two pieces so downstream systems (billing, reporting, frequency capping) get complete information.
Simple Analogy
Imagine ordering food delivery. The restaurant (ad server) knows every detail about your order. Your doorbell ring (client event) only signals arrival. Enrichment matches the doorbell ring to the specific order details.
Step 2: Understand Why the Old Design Failed
The Old Architecture
Ad Server ──writes──► Metadata Registry (Key-Value Store)
│
Client Event ──reads──────────┘
│
▼
Enriched Ad Event
Why This Worked Initially
- Globally replicated → any region could find metadata
- Simple and reliable for normal streaming (SVOD)
Why Live Events Broke It
Live events create extreme traffic spikes:
Normal SVOD Traffic: ────────────────────────
Live Event Traffic: ─────────▲▲▲▲▲──────────
(100M+ viewers
simultaneously)
The math problem:
| Scale | Impact |
|---|---|
| 100M+ concurrent viewers | Millions of ad events per second |
| Each event needs metadata written first | Millions of DB writes/second on critical path |
| DB write must succeed before ad serves | Single point of failure |
Three core problems created:
- 🔴 Scalability — database cannot handle write volume
- 🔴 Reliability — DB outage stops ad serving entirely
- 🔴 Latency — waiting for DB acknowledgment slows serving
Step 3: Understand the Core Insight
The Key Question Netflix Asked
"The ad server already logs everything. Do we really need to write to a database BEFORE serving? Can we match streams AFTER the fact?"
The Shift in Thinking
OLD APPROACH (synchronous):
Ad Server → Write to DB → Serve Ad → Client fires event → Read from DB → Enrich
NEW APPROACH (asynchronous):
Ad Server → Serve Ad → Log response to stream ──┐
├──► Join in Flink → Enrich
Client fires event → arrives on stream ──────────┘
Key benefit: The database write is completely off the critical path
Step 4: Learn How the Flink Streaming Join Works
What is Apache Flink?
Apache Flink is a framework built specifically for stateful stream processing — processing continuous data streams while remembering state between events.
The New Architecture
Ad Serving Response Stream ──────────────────────┐
▼
[Flink Streaming Job]
│
Client Event Stream ──────────────────────────────┘
│
▼
Enriched Event Stream
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
Billing Reporting Frequency Cap
The 60-Minute Window
Flink holds serving responses in memory for 60 minutes, waiting for matching client events:
Timeline:
0:00 ── Ad served, response logged ──────────────────────────────────────►
0:03 ── Client fires "impression" event → MATCHED immediately ✓
0:15 ── Client fires "first quartile" event → MATCHED ✓
0:45 ── Client fires "completion" event → MATCHED ✓
1:05 ── Late event arrives → MISSED (goes to recovery path)
Three Outcomes for Every Event
Client Event Arrives
│
▼
Find matching
serving response?
│
┌────┴────┐
YES NO
│ │
▼ ▼
Enrich Still within YES → Hold, wait for response
immediately 60 min? ──►
│
NO
▼
Send to recovery
(Spark batch job)
Why Custom Operator Instead of Built-in Join?
Netflix built a KeyedCoProcessFunction (custom operator) because:
| Built-in Windowed Join | Custom Operator |
|---|---|
| Enforces time ordering | Not needed (downstream handles ordering) |
| Fixed semantics | Full control over matching logic |
| Hard to tune retention | Tune per stream type |
| Cannot easily extend | Can add features without rewriting |
Step 5: Understand Cross-Region Routing
The Problem
Region A (us-east-2) Region B (us-east-1)
──────────────────── ────────────────────
Ad served here Member switches from
Response stored here wifi → mobile data
Event arrives HERE
❌ No matching response!
Solution: Route the Event, Not the Data
Rejected approach:
❌ Replicate ALL responses to ALL regions
→ Wasteful: 99%+ of responses never used cross-region
→ High cost and compute
Chosen approach:
✅ Encode serving region INTO the ad token
Token contains: [ad_id | serving_region | other_data]
│
▼
Event arrives in wrong region → Read token → Forward to correct region
us-east-1 receives event → reads token → "belongs to us-east-2" → forwards it
Flow:
Device ──event──► us-east-1
│
│ reads token: "served in us-east-2"
│
▼
us-east-2 ──► matches response ──► enriched event
Only cross-region events pay the extra hop cost — the majority are handled locally.
Step 6: Understand Late Event Recovery
Why a 60-Minute Window Isn't Enough for Everything
Some events arrive very late due to:
- Device was offline
- Ad was paused for a long time
- Network issues
- Certain engagement event types
The Hybrid Architecture
┌─────────────────────────────────┐
│ Flink Streaming Job │
│ (handles ~99.999% of events) │
│ Window: 60 minutes │
└─────────────────────────────────┘
│
Unmatched after 60 min
│
▼
┌─────────────────────────────────┐
│ Dead Letter Queue / Storage │
└─────────────────────────────────┘
│
Hourly trigger
│
▼
┌─────────────────────────────────┐
│ Spark Batch Job (Maestro) │
│ Matches against data warehouse │
│ (much longer retention) │
└─────────────────────────────────┘
│
▼
Same downstream sinks as streaming path
This batch job also handles:
- Replaying dead-letter queues
- Backfilling historical data
- Operational recovery after incidents
Step 7: Understand Deduplication
Why Duplicates Happen
Client device retries event ──────────────────────┐
├──► Same event arrives twice
Upstream at-least-once delivery ───────────────────┘
For billing, counting the same event twice = charging advertisers incorrectly.
The Solution: Stable Idempotent Key
Event fields:
┌─────────────────────────────────────────────┐
│ ad_id + event_type + timestamp/offset │
└─────────────────────────────────────────────┘
│
▼ hash function
stable_event_id
Same underlying event → always same ID → second copy is dropped
Old vs. New Deduplication
| Aspect | Old (External KV Store) | New (Flink Local State) |
|---|---|---|
| Where IDs stored | External database | Local to Flink job |
| Network hop | Yes, every event | No |
| Race conditions | Possible | Eliminated (keyed routing) |
| Speed | Slower | Faster |
Important Guarantee Distinction
Netflix's guarantee:
✅ At-least-once delivery + best-effort deduplication
❌ NOT exactly-once (too expensive, not necessary)
Why acceptable:
• In-stream dedup removes vast majority
• Event ID travels with enriched event
• Billing systems run their own dedup over longer windows
Step 8: Understand the Migration Strategy
The Challenge
Migrating a live, revenue-critical pipeline = like replacing an airplane engine mid-flight
Phase 1: Parallel Running
All Events
│
├──────────────────────────────────────────────┐
│ │
▼ ▼
Legacy Pipeline Flink Pipeline
│ │
▼ ▼
Output A Output B
│ │
└──────────────────┬───────────────────────────┘
│
▼
Automated Audit
(alerts on drift)
Shared enrichment logic module = both pipelines use identical enrichment code → any difference in audit = real pipeline bug, not logic difference
Phase 2: Deterministic Traffic Split
❌ Random split:
Event X → Legacy (published)
Event X → Flink (also published)
= DOUBLE COUNTED downstream
✅ Deterministic split using stable key:
Event X → stable_key → always routes to ONE pipeline
= Published exactly once
Cutover dial:
0% ──────────────────────────────────────────► 100%
[Legacy handles all] [Flink handles all]
│
└── Raise gradually, monitor, can roll back instantly
Phase 3: Validation and Decommission
Agreement rate: >99.99% ✓
Region-by-region rollout ✓
Legacy service decommissioned ✓
Metadata Registry deleted ✓
Single point of failure: GONE ✓
Step 9: Key Engineering Principles Illustrated
Principle 1: Keep the Hot Path Clean
❌ Tempting: Add extra lookups, side processing to Flink job
✅ Correct: Job's only job = matching events
Everything else → off the hot path
Principle 2: Design for Operational Reality
| Concern | Solution |
|---|---|
| Checkpoint performance | Trim state, work with platform team |
| Deployment safety | Canary deployments, keep old job until new one healthy |
| Observability | Custom canary metrics, CI/CD guardrails |
Principle 3: Build Reusable Recovery Mechanisms
Spark batch job serves multiple purposes:
├── Long-tail event recovery (>60 min)
├── Dead-letter queue replay
├── Historical backfill
└── Operational incident recovery
Summary: The Complete Picture
BEFORE: AFTER:
Ad Server Ad Server
│ │
▼ ▼
Write to ──────────────────► Log to Kafka stream
Metadata Registry (❌ critical path) │
│ │
▼ ▼
Client Event ──► Read Registry ┌─── Flink Job ───┐
│ │ 60-min window │
▼ │ Stateful join │◄── Client Events
Enriched Event │ Local dedup │
└─────────────────┘
│
┌───────┴────────┐
│ │
▼ ▼
Enriched Event Unmatched →
(real-time) Spark Recovery
What was gained:
- ✅ Removed single point of failure from critical path
- ✅ Elastic scaling for Live events
- ✅ Faster, local deduplication
- ✅ Cross-region routing without full replication
- ✅ Zero-downtime migration of revenue-critical system