After studying this material, you should be able to:

1. Explain the training-serving skew problem and why feature consistency matters in ML systems
2. Identify the root causes of feature inconsistency between online and offline pipelines
3. Describe the feature logging framework as a solution architecture
4. Understand the scale challenges and the specific engineering techniques used to address them
5. Evaluate trade-offs in distributed stream processing systems

---

Step-by-Step Teaching

Step 1: The Core Problem — Training-Serving Skew

What Is It?

Imagine teaching someone using one language, then testing them in a slightly different dialect. They will underperform — not because they lack knowledge, but because the format changed.

This is exactly what happens in ML systems.

Training Data:          Serving (Inference) Data:
language = "en-US"  ≠  language = "en"
region   = "jp_JA"  ≠  region   = "jp-JA"

The model learned patterns from one format. At prediction time, it receives a different format. The model silently degrades.

Why Is It Silent?

There is no crash, no error message. The system keeps running. But:

• Feature distributions shift
• Prediction quality drops
• Engineers spend weeks debugging ETL pipelines to find the cause

The Two Pipelines That Must Match

ONLINE PATH (Inference/Serving)
─────────────────────────────────────────────────────
User Request → Feature Store → Model → Prediction
                    ↑
              Real-time features
              (user_id, language, store metadata)

OFFLINE PATH (Training)
─────────────────────────────────────────────────────
App Logs + Hive Tables + Click Logs
         ↓
   ETL Transformations
         ↓
   Training Dataset → Model Training

The problem: These two paths compute features independently, introducing subtle mismatches.

Additional Offline Problems

---

Step 2: The Solution — Feature Logging

Core Idea

Log the exact features used at inference time, then use those same logged features to train the next model.

BEFORE (inconsistent):
Online Pipeline  →  Features A  →  Prediction
Offline Pipeline →  Features B  →  Training
                         ↑
                    A ≠ B = Problem

AFTER (consistent):
Online Pipeline  →  Features A  →  Prediction
                         ↓
                    Log Features A
                         ↓
Offline Pipeline →  Features A  →  Training
                         ↑
                    A = A = ✓

This creates a single source of truth: the features used for prediction become the features used for training.

---

Step 3: Scale Challenges — Why This Is Hard

Feature logging sounds simple. At Uber's scale, it becomes an enormous engineering challenge.

Challenge 1: Volume

Traffic:     8,000,000 predictions per second (8M QPS)
Daily rows:  ~100 billion records
Monthly:     ~5 trillion rows
Storage:     ~1.7 PB per day

Logging everything naively would cost millions of dollars in infrastructure.

Challenge 2: Verbose Feature Names

Feature names are long and descriptive:

store_unique_identifier_operational_meal_period_context_key

Sending this string billions of times per day across the network is extremely wasteful.

Estimated outbound data rate: ~9.3 GiB/second
Kafka cluster capacity: far exceeded

Challenge 3: Unnecessary Features

Clients send more features than the model actually uses. Logging all of them wastes bandwidth and storage.

Challenge 4: Not All Predictions Are Useful

In a recommendation system:

Candidates scored:          1,000
After filtering/ranking:      400  (40%)
Actual user impressions:       20  (5% of ranked)

Logging all 1,000 candidates when only 20 matter wastes 98% of storage on data that adds little training value.

---

Step 4: Engineering Solutions — How Each Challenge Was Solved

Solution 1: Feature Allow List

Only log features the model actually uses.

Before: Log all 200 features in request payload
After:  Log only 40 features the model needs

Result: 4–5× reduction in payload size

Solution 2: Feature Name Aliasing via Enums

Replace long string names with compact integer IDs.

Before: "store_unique_identifier_operational_meal_period_context_key"
After:  42

Transmitted billions of times → massive bandwidth savings

The mapping is deterministic, so the integer can always be decoded back to the original name.

Solution 3: Impression Filtering with Apache Flink

Only log predictions that actually became user impressions.

Stream 1: Prediction events (from inference)
Stream 2: Client impression events (from user devices)
                    ↓
              Flink Stream Join
              (time-windowed, few minutes)
                    ↓
         Only matched records are logged

Key engineering details:

Solution 4: Handling Transformer Model Sequence Features

Transformer models output arrays of candidates, not single rows:

store_uuid: [taco_store, McD, burger_store, ...]

Sending arrays directly breaks Kafka payload limits and complicates impression filtering.

Solution: Flatten arrays into individual key-value records before sending to Kafka.

Before: {store_uuid: [taco_store, McD, burger_store]}
After:  {store_uuid: taco_store}
        {store_uuid: McD}
        {store_uuid: burger_store}

Additionally, data is sharded across multiple Kafka clusters in round-robin fashion to distribute load.

---

Step 5: Lessons in Distributed Systems Engineering

These lessons apply broadly beyond this specific system.

Lesson 1: Profiling Before Scaling

Increasing parallelism alone does not solve bottlenecks.

Each operator in a pipeline has unique scaling characteristics. Profile first, then tune each stage independently based on data.

Lesson 2: Observability Enables Optimization

Before tuning, invest in detailed metrics:

• Job output rates
• Time-window distributions
• Pipeline behavior

This replaces trial-and-error with data-driven engineering.

Lesson 3: Throughput Comes from Many Small Improvements

No single optimization solved the performance problem. The gains came from combining:

Object reuse
+ Typed payloads
+ Configuration caching
+ Reduced serialization overhead
─────────────────────────────
= ~70% reduction in peak consumer lag

Lesson 4: Build Validation and Automation In

Reliable systems require:

• Automated rollbacks
• Alerts-as-code
• Deterministic validation queries
• Production-like testing environments

---

Step 6: Results — Did It Work?

---

Summary: The Complete Picture

PROBLEM
Training and serving pipelines compute features independently
→ Silent mismatches → Model degradation

SOLUTION
Log exact inference features → Use them for training
→ Single source of truth

SCALE CHALLENGES & SOLUTIONS
┌─────────────────────────┬──────────────────────────────┐
│ Challenge               │ Solution                     │
├─────────────────────────┼──────────────────────────────┤
│ Massive data volume     │ Impression filtering (Flink) │
│ Verbose feature names   │ Enum aliasing                │
│ Unnecessary features    │ Feature allow list           │
│ Transformer arrays      │ Flatten + Kafka sharding     │
│ State management        │ Custom RocksDB strategy      │
└─────────────────────────┴──────────────────────────────┘

OUTCOME
0% mismatch, hours-fresh features, sustainable infrastructure cost

---

Self-Check Questions

1. What is training-serving skew and why is it dangerous?
2. Why does logging every prediction become impractical at 8M QPS?
3. How does enum aliasing reduce bandwidth without losing information?
4. Why is impression filtering more efficient than logging all predictions?
5. What problem does flattening transformer array features solve?
6. Why is profiling individual operators important in a Flink pipeline?