Identified Learning Outcomes

After studying this material, students should be able to:

1. Understand how large-scale ML systems evolve in production ad systems
2. Apply sequence modeling techniques for user behavior prediction
3. Explain the challenges of online-offline discrepancy in ML systems
4. Describe multi-task learning architectures for ads ranking

---

Step-by-Step Teaching Guide

STEP 1: Foundation Concepts

What is Ads Engagement Modeling?

User Sees Ad → User Acts (Click/Save/Buy) → System Learns → Better Predictions

Core Problem:

Predict the probability that a user will engage with an advertisement

Key Metrics:

• CTR (Click-Through Rate)
• CVR (Conversion Rate)
• Engagement Rate

---

STEP 2: Evolution of Models

From Simple to Complex (GBDT → DNN)

2.1 GBDT (Gradient Boosted Decision Trees)

Advantages          |  Limitations
--------------------|------------------
✓ Interpretable     |  ✗ Manual features
✓ Fast training     |  ✗ Limited patterns
✓ Works on tabular  |  ✗ No sequence data

2.2 Deep Neural Networks (DNN)

Input Features
     ↓
[Embedding Layer]  ← Converts categories to vectors
     ↓
[Hidden Layers]    ← Learns complex patterns
     ↓
[Output Layer]     ← Probability prediction

2.3 Why the Evolution?

More Data Available
        +
Complex User Behaviors
        +
Need for Better Accuracy
        =
Move to Deep Learning

---

STEP 3: User Action Sequence Modeling

Understanding Sequential Behavior

3.1 Core Concept

User Journey Example:
[Saw Recipe Pin] → [Saved Recipe] → [Clicked Kitchen Ad] → [Bought Item?]
     t-3               t-2               t-1                    t

Key Insight:

Past user actions contain valuable signals about future behavior

3.2 Technical Components

A) Sequence Representation

# Conceptual representation
user_sequence = {
    'actions': ['view', 'save', 'click'],
    'items':   ['pin_1', 'pin_2', 'ad_1'],
    'timestamps': [t1, t2, t3]
}

B) Attention Mechanism

Recent Actions    →  Higher Weight  →  More Influence
Older Actions     →  Lower Weight   →  Less Influence

[Action_1] × 0.1
[Action_2] × 0.3
[Action_3] × 0.6  ← Most Recent
                  ↓
            Final Prediction

3.3 Why Sequences Matter

---

STEP 4: Multi-Task Learning Architecture

Learning Multiple Goals Simultaneously

4.1 Basic Architecture

                    [Shared Layers]
                    /      |      \
                   /       |       \
            [Task 1]   [Task 2]  [Task 3]
            Click       Save     Purchase
            Pred.       Pred.     Pred.

4.2 Why Multi-Task?

Problem: Purchase data is SPARSE
         (Few users actually buy)

Solution: Learn from related tasks
         
Clicks (Abundant) ──┐
Saves  (Medium)   ──┼──► Better Purchase Prediction
Views  (Abundant) ──┘

4.3 Ensemble Approach

Model 1 (Specialized)  ─┐
Model 2 (Specialized)  ─┼──► Ensemble ──► Final Score
Model 3 (Specialized)  ─┘

---

STEP 5: Online-Offline Discrepancy

The Critical Production Challenge

5.1 What is the Problem?

OFFLINE (Training)          ONLINE (Production)
─────────────────          ──────────────────
Historical data      ≠      Real-time data
Static distribution  ≠      Dynamic distribution
Controlled testing   ≠      Real user behavior

5.2 Common Causes

1. DATA FRESHNESS
   Training Data (Old) ──► Model ──► Predicts on New Data
                                     ↑
                              Distribution Shift!

2. FEATURE INCONSISTENCY
   Offline Features ≠ Online Features
   (Batch computed)   (Real-time computed)

3. FEEDBACK DELAY
   User Buys Item
        ↓
   [Hours/Days Later]
        ↓
   System Records Conversion
        ↓
   Model Doesn't Know Yet!

5.3 Solutions Framework

Problem              Solution
─────────────────────────────────────────
Data Freshness    →  Continuous Training
Feature Gap       →  Feature Store Alignment
Feedback Delay    →  Delayed Feedback Models
Distribution Shift→  Online Learning

---

STEP 6: Complete System Architecture

Putting It All Together

USER BEHAVIOR
     │
     ▼
┌─────────────────────────────────┐
│     FEATURE ENGINEERING         │
│  • User Sequences               │
│  • Ad Features                  │
│  • Context Features             │
└─────────────────────────────────┘
     │
     ▼
┌─────────────────────────────────┐
│     MULTI-TASK DNN              │
│  • Shared Representation        │
│  • Task-Specific Layers         │
│  • Ensemble Components          │
└─────────────────────────────────┘
     │
     ▼
┌─────────────────────────────────┐
│     ONLINE SERVING              │
│  • Real-time Inference          │
│  • Discrepancy Monitoring       │
│  • Continuous Updates           │
└─────────────────────────────────┘
     │
     ▼
   AD RANKING SCORE

---

Key Takeaways Summary

---

Self-Assessment Questions

1. Basic: Why did Pinterest move from GBDT to DNN models?

2. Intermediate: How does attention mechanism help in sequence modeling?

3. Advanced: Design a solution for handling delayed conversion feedback in an ads system

4. Applied: What metrics would you monitor to detect online-offline discrepancy?