Step-by-Step Teaching

STEP 1: The Problem — Why Traditional Search Breaks

What is Lexical Search?

Lexical search matches exact strings of text. It looks for the literal words you typed.

Query: "soda"
Document: "soft drink"
Lexical result: ❌ NO MATCH (different words, same meaning)

Real-World Failure Cases

Why This Matters for Uber Eats

When search misses intent → users bounce (leave without ordering) When search understands intent → users find what they want in seconds

Key Insight: Search is not just a feature — it directly drives revenue through conversion rates.

---

STEP 2: The Solution — Semantic Search

Core Concept: Meaning Over Words

Semantic search converts both queries and documents into vectors (lists of numbers) in the same mathematical space.

"soda"       → [0.2, 0.8, 0.1, ...]
"soft drink" → [0.21, 0.79, 0.11, ...]
              ↑ These are CLOSE in vector space = same meaning ✅

Visual Analogy

LEXICAL SPACE          SEMANTIC SPACE
(string matching)      (meaning matching)

"soda" ----X---- "soft drink"    "soda" ≈ "soft drink" ✅
(no connection)                   (close together)

What Semantic Search Enables

• Cross-language understanding ("pan" → bread in Spanish context)
• Typo tolerance ("mozzarela" → mozzarella)
• Shorthand understanding ("gf" → gluten-free)
• Context awareness

---

STEP 3: The Architecture — Two-Tower Model

Why Two Towers?

The two-tower design separates query processing from document processing. This is the most important architectural decision.

┌─────────────────────────────────────────────────────┐
│                  TWO-TOWER MODEL                     │
│                                                      │
│  QUERY TOWER              DOCUMENT TOWER             │
│  ┌──────────┐             ┌──────────┐               │
│  │"gf pizza"│             │"Gluten   │               │
│  │          │             │Free      │               │
│  │  QWEN    │             │Margherita│               │
│  │  LLM     │             │  QWEN    │               │
│  │ backbone │             │  LLM     │               │
│  └────┬─────┘             │ backbone │               │
│       │                   └────┬─────┘               │
│  [0.2, 0.8, ...]          [0.21, 0.79, ...]          │
│       └──────────────┬─────────┘                     │
│                 Similarity Score                      │
└─────────────────────────────────────────────────────┘

The Critical Advantage: Decoupling

Key Insight: You only compute document embeddings once (or periodically), not for every search. This makes the system scalable to billions of items.

The Backbone: QWEN LLM

• Large Language Model used as the base layer
• Brings world knowledge (knows "gf" means gluten-free)
• Brings cross-lingual ability (understands Spanish "pan" = bread)
• Fine-tuned on Uber Eats data to adapt to food delivery context

---

STEP 4: Training the Model

The Loss Function: InfoNCE with MRL

InfoNCE Loss trains the model by:

• Pulling matching query-document pairs closer in vector space
• Pushing non-matching pairs further apart

Training signal:
Query: "gf pizza"
✅ Positive: "Gluten Free Margherita" → push CLOSER
❌ Negative: "Chocolate Cake" → push FURTHER

Matryoshka Representation Learning (MRL)

Named after Russian nesting dolls 🪆 — smaller versions fit inside larger ones.

Full embedding (1536 dimensions):
[d1, d2, d3, d4, ... d256, ... d512, ... d1536]
 └──────────┘  └─────────────┘  └────────────────┘
  128-dim cut    256-dim cut       512-dim cut
  (fast, coarse) (balanced)        (detailed)

Why this matters: One model, multiple precision levels. You choose the tradeoff at serving time.

Training Infrastructure

PyTorch + Hugging Face Transformers
         ↓
Ray (distributed orchestration)
         ↓
DDP + DeepSpeed ZeRO-3
(shards optimizer, gradients, model parameters across GPUs)
         ↓
Trains on hundreds of millions of data points

---

STEP 5: Serving at Scale — The Index

The Challenge

Uber Eats has billions of menu items across restaurants, grocery, and retail. For every search query, you need to find the most similar items in milliseconds.

The Solution: ANN (Approximate Nearest Neighbor) Search

Exact nearest neighbor search across billions of vectors is too slow. ANN finds approximately the best matches, much faster.

Exact Search:  Compare query to ALL 1 billion items → too slow ❌
ANN Search:    Smart graph traversal → finds top matches in ms ✅

HNSW Graph (the index structure)

HNSW (Hierarchical Navigable Small World) builds a layered graph:

Layer 2 (coarse):  A ──────────── E
                   |              |
Layer 1 (medium):  A ── B ── C ── E
                   |    |    |    |
Layer 0 (fine):    A─B─C─D─E─F─G─H  ← all documents here

Search starts at the top (fast, coarse) and drills down to precise results.

Apache Lucene Plus Integration

• Separate indices for each vertical: OFD (restaurants) vs GR (grocery/retail)
• Combines vector search with traditional filters
• Pre-filters run before ANN search to shrink candidate set

Search Flow:
User Query
    ↓
Pre-filters (city, location hexagon, food type, fulfillment)
    ↓ [billions → thousands of candidates]
ANN Search (HNSW)
    ↓ [thousands → top-k results]
Optional micro re-ranking (compact neural network)
    ↓
Downstream ranker
    ↓
Final results shown to user

---

STEP 6: Cost Optimization — The Three Levers

This is where engineering meets economics. Three knobs to tune:

Lever 1: ANN Parameter K (number of neighbors per shard)

High K (k=1200): More candidates retrieved → higher recall, higher cost
Low K  (k=200):  Fewer candidates retrieved → similar recall, lower cost

Result: 34% latency reduction + 17% CPU savings with negligible recall loss

Why does this work? Pre-filters already narrow the candidate set, so you don't need as many neighbors per shard to maintain quality.

Lever 2: Quantization (number precision)

float32 (fp32): Each number = 32 bits → high precision, high storage
int7 (SQ):      Each number = 7 bits  → slight precision loss, much smaller

Result: >50% latency reduction, recall stays above 0.95

Visual:
fp32:  [0.23847261, 0.71923847, ...]  ← precise but large
int7:  [30, 92, ...]                  ← approximate but tiny

Lever 3: MRL Embedding Dimension

Full (1536 dim): Maximum quality, maximum storage
Cut  (256 dim):  <0.3% quality loss for English/Spanish, 50% storage reduction

Combined Impact

Key Insight: These optimizations stack — apply all three for dramatic cost reduction with minimal quality impact.

---

STEP 7: Production Reliability — Keeping It Running

The Freshness Problem

Uber's data changes continuously — new restaurants open, menus change, items go out of stock. The search index must stay fresh.

Solution: Automated end-to-end refresh every 2 weeks

Biweekly Pipeline:
Train/fetch model → Generate embeddings → Build new index → Validate → Deploy

Blue/Green Deployment

The most critical reliability mechanism. Think of it like a light switch with a backup.

Single Index with Two Columns:
┌─────────────────────────────────────┐
│           SEARCH INDEX              │
│                                     │
│  embedding_BLUE  │ embedding_GREEN  │
│  (Model v1.0)    │ (Model v2.0)     │
│                  │                  │
│  ← ACTIVE        │  ← INACTIVE      │
└─────────────────────────────────────┘
         ↑
   Config maps "active" to BLUE

During refresh:

1. Keep BLUE serving live traffic (untouched)
2. Rebuild GREEN with new model embeddings
3. Validate GREEN thoroughly
4. Switch config: GREEN becomes active
5. BLUE becomes the rollback option

Before refresh:  BLUE=active (v1.0),  GREEN=inactive (v1.0)
After refresh:   BLUE=inactive (v1.0), GREEN=active (v2.0)
If problem:      Switch back → BLUE=active (v1.0) instantly

Three Validation Gates (Must Pass Before Deployment)

Gate 1: COMPLETENESS
"Are all documents present?"
→ Count documents in new index vs. source tables
→ Fail if counts don't match

Gate 2: BACKWARD COMPATIBILITY  
"Did we accidentally change the active column?"
→ Compare active column byte-for-byte between old and new build
→ Any difference = abort deployment

Gate 3: CORRECTNESS
"Is the new model actually better (or at least equal)?"
→ Deploy to non-production environment
→ Replay real queries
→ Compare recall vs. current production
→ Must meet quality threshold to proceed

Serving-Time Safety Net

Even after deployment, the system continuously self-checks:

For sampled requests:
  Query embedding model ID == Index column model ID?
  
  ✅ Match → serve normally
  ❌ Mismatch → emit error metric
  
  If errors persist for X minutes:
    → Auto-rollback model to previous version
    → Alert engineers
    → Index unchanged (safe)

Why this matters: Catches human errors like wrong config mappings without adding latency to every request.

---

STEP 8: Putting It All Together

Complete System Flow

USER TYPES: "gf pizza near me"
                ↓
        QUERY TOWER (real-time)
        QWEN LLM → 256-dim vector
                ↓
        PRE-FILTERS
        (city_id, location, fulfillment_type)
        Billions → Thousands of candidates
                ↓
        ANN SEARCH (HNSW, k=200, int7 quantized)
        Thousands → Top-k results
                ↓
        MICRO RE-RANKING (compact neural net)
                ↓
        DOWNSTREAM RANKER
                ↓
USER SEES: "Gluten Free Margherita", "GF Pepperoni", ...

System Properties Summary

---

Key Takeaways

1. Semantic search solves lexical search's fundamental limitation by matching meaning, not strings
2. Two-tower architecture enables scale by decoupling query (real-time) from document (batch) processing
3. MRL + quantization + K-tuning are three independent levers that together dramatically reduce cost with minimal quality loss
4. Blue/green deployment with automated validation gates is what makes continuous model updates safe in production
5. Pre-filters before ANN is a critical latency optimization — shrink the candidate set before the expensive vector search