After studying this material, you should be able to:

1. Explain why organizations choose in-house LLM serving over hosted APIs
2. Describe the four core architectural decisions in building an LLM serving platform
3. Identify production challenges that only emerge under real load
4. Understand how constrained decoding works and why it matters
5. Apply these architectural patterns to evaluate or design similar systems

---

Step-by-Step Study Material

Step 1: The Big Picture — Why Run LLMs In-House?

The Default Approach

Most organizations use hosted APIs (like OpenAI's API):

Your App → API Call → OpenAI Servers → Response

Netflix's Approach

Netflix runs the full stack internally:

Your App → Netflix's own routing → Netflix's own GPU servers → Response

Why bother?

Key Insight: The payoff is that moving from a hosted model to a self-hosted fine-tuned model requires minimal code changes — same API, different backend.

---

Step 2: Understanding the Infrastructure Architecture

Before the four decisions, you need to understand the existing system Netflix built upon.

The Serving Stack (Bottom to Top)

┌─────────────────────────────────────────┐
│         Downstream Applications         │
└────────────────┬────────────────────────┘
                 │
    ┌────────────┴────────────┐
    │   gRPC Path             │   HTTP Path (newer LLM apps)
    │   (JVM Serving System)  │
    └────────────┬────────────┘
                 │
    ┌────────────▼────────────┐
    │  Model Scoring Service  │  ← Shared inference backend
    │  (MSS)                  │    Supports XGBoost, TF, PyTorch, LLMs
    └────────────┬────────────┘
                 │
    ┌────────────▼────────────┐
    │  NVIDIA Triton          │  ← Manages model loading,
    │  Inference Server       │    batching, GPU scheduling
    └────────────┬────────────┘
                 │
    ┌────────────▼────────────┐
    │  Java Control Plane     │  ← Deployment, versioning,
    │                         │    health checks, autoscaling
    └─────────────────────────┘

Where Models Actually Run

• Small CPU models → Run in-process (no network call overhead)
• Large GPU models → Delegated to MSS remotely

Analogy: Think of MSS like a shared kitchen in a restaurant. Individual stations (applications) prep ingredients locally, but all cooking on the big stoves (GPUs) happens in the shared kitchen.

---

Step 3: The Four Core Decisions

These decisions are presented in dependency order — each one constrains the next.

---

Decision 1: Engine Selection

The Question: Which inference engine should power LLM serving?

Original Choice: TensorRT-LLM

• Already integrated with Triton
• High performance at the time

Why They Switched to vLLM

By mid-2025, two things changed:

1. Performance gap closed — Open-source engines caught up to specialized stacks
2. Workload diversified — They now needed to handle:

┌─────────────────────────────────────────────┐
│ Workload Types                              │
├─────────────────────────────────────────────┤
│ • Embedding generation                      │
│ • Prefill-only inference (ranking/retrieval)│
│ • Autoregressive decoding (text generation) │
│ • Custom models with per-step constraints   │
└─────────────────────────────────────────────┘

vLLM won on operational fit, not just raw speed.

Key Concept — Operational Fit: A system isn't just about peak performance. It's about how well it integrates, how easy it is to maintain, and how well it handles your specific workload mix.

---

Decision 2: Model Packaging

The Question: How do you package models for vLLM inside Triton?

Triton offers two packaging approaches. The core trade-off is:

Tight Coupling          vs.          Loose Coupling
─────────────────                    ──────────────
Model artifacts tied                 Model artifacts
to frontend version                  independent of
                                     frontend version

The Production Problem They Hit:

When model artifacts are tightly coupled to the frontend, upgrading the serving engine requires repackaging every model — a maintenance burden that grows with the number of models.

Lesson: Packaging decisions that seem minor in development become significant maintenance costs in production at scale.

---

Decision 3: API Surface Design

The Question: How should callers interact with the LLM serving system?

Design Goal: No Special Snowflakes

Netflix's principle: Every model — XGBoost or LLM — is scored via the same interface.

This means:

• Same client libraries
• Same health checking
• Same deployment pipelines

The Dual API Strategy

Callers
  │
  ├──► gRPC (existing path — all model types)
  │
  └──► HTTP / OpenAI-compatible API (new path — LLM ecosystem)

Why OpenAI-compatible API?

The OpenAI API has become the de facto standard. This means:

• Inference engines speak it
• Orchestration frameworks speak it
• Evaluation tools speak it
• Client libraries speak it

Adopting it means Netflix gets ecosystem compatibility for free.

The Production Bug They Found

Problem:
Caller sends request with response_format: JSON
         ↓
OpenAI-compatible frontend accepts it ✓
         ↓
response_format silently dropped before reaching vLLM ✗
         ↓
vLLM generates without JSON constraints
         ↓
Caller receives malformed JSON with NO error

Fix: They patched the frontend (using git-subtree) to translate response_format into vLLM's guided decoding parameters.

Key Lesson: "Accepted by schema" ≠ "actually enforced." Silent failures are more dangerous than loud ones.

---

Decision 4: Deployment / Rollout Strategy

The Question: How do you update models without dropping requests?

The Challenge: GPU deployments take longer to start than CPU services, and the I/O schema may change between versions.

Two Rollout Strategies

Strategy A: Red-Black Deployment

Old Version (Red) ──► Still serving traffic
New Version (Black) ──► Spins up fully
                         ↓
                    Traffic switches atomically
                         ↓
Old version terminates

• ✅ Cheaper
• ✅ Simpler
• ⚠️ Requires version-agnostic model (no schema changes)

Strategy B: Versioned Deployment

Old Version ──► Continues serving old schema
New Version ──► Serves new schema simultaneously
                 ↓
            Gradual traffic migration
                 ↓
Old version retires

• ✅ Handles breaking interface changes
• ❌ More expensive (two versions running simultaneously)
• ❌ More complex coordination

Netflix's Recommendation:

Embed variable configurations (like tensor shapes) directly into the model → use Red-Black (cheaper). Reserve Versioned for unavoidable breaking changes.

---

Step 4: Operational Details That Surprised Them

These are the "design phase didn't anticipate" lessons — arguably the most valuable part.

Surprise 1: Boot Sequence Complexity

Bringing up a vLLM-on-Triton instance requires coordinated steps:

Boot Sequence:
1. Extract model package
2. Install custom vLLM plugins (via Python entry_points)
3. Clean Prometheus multiprocess directory
4. [Non-routine steps involving model loading]
5. Gate gRPC port until engine is READY

The gRPC port only opens after the engine is fully ready — preventing traffic from hitting an uninitialized instance.

---

Surprise 2: Observability Gap

The Problem:

vLLM metrics ──► Written to disk as .db files
Triton metrics ──► Exposed via its own Prometheus endpoint

Neither knows about the other.
Triton's bridge only surfaces 9 of 40+ vLLM metrics.

Missing: token throughput, KV cache utilization, 
         prefix cache hit rates

The Fix: A lightweight HTTP proxy that merges both:

/metrics endpoint
      │
      ├── Fetches Triton metrics via HTTP
      └── Reads vLLM metrics from disk (MultiProcessCollector)
              │
              └── Returns combined output
                  (existing dashboards work unchanged)

Key Lesson: When integrating two systems, observability gaps are invisible until production. Always audit what metrics are actually being collected vs. what you think is being collected.

---

Step 5: Constrained Decoding — Pushing Logic Inside the Model

The Problem It Solves

Without constraints:

Model generates output
    ↓
Business logic checks if output is valid
    ↓
If invalid → retry or repair
    ↓
Wasted compute + added latency

With constrained decoding:

At each token generation step:
    ↓
Constraint logic runs INSIDE the decode loop
    ↓
Only valid tokens are eligible
    ↓
Output is compliant by construction

How It Works: State Machine + Token Masks

Request arrives with constraints
    ↓
Each request gets its own logits processor
    ↓
At each decode step:
    State Machine evolves based on tokens generated so far
         ↓
    Emits a token-eligibility mask
         ↓
    Only eligible tokens can be selected
         ↓
    Next token generated

Analogy: Imagine autocomplete that physically cannot suggest words that would make your sentence grammatically wrong — not just unlikely, but impossible.

---

Step 6: The Scaling Problem and Its Evolution

vLLM V0: The CPU Bottleneck

GPU: Produces logits for entire batch (fast, parallel)
         ↓
CPU: Copies logits from GPU (transfer overhead)
         ↓
CPU: Runs constraint logic for Request 1
CPU: Runs constraint logic for Request 2  ← Sequential!
CPU: Runs constraint logic for Request 3  (GIL prevents parallelism)
...

Result: CPU time grows linearly with batch size.

Batch Size:  1    10    50    100
CPU Time:    1x   10x   50x   100x  ← Linear scaling = bad

This bottleneck is invisible in single-request benchmarks — it only appears under realistic concurrency.

vLLM V1: The Fix

GPU: Produces logits for entire batch
         ↓
CPU: Processes ALL requests' constraints TOGETHER (batch-level)
     Implemented in C++ with multi-threading (bypasses GIL)
         ↓

Result: CPU time stays roughly flat as batch size grows.

Batch Size:  1    10    50    100
CPU Time:    1x   1.2x  1.5x  1.6x  ← Flat scaling = good

Trade-off: V1's API is more complex — requires explicit tracking of batch membership changes via update_state(batch_update).

---

Two New Problems That Emerged in V1

Even after solving the performance problem, stateful constraint logic introduced new issues:

Problem 1: State Consistency

• Batch membership changes dynamically (requests finish, new ones join)
• Constraint state machines must stay synchronized with actual batch state
• A mismatch = wrong constraints applied to wrong request

Problem 2: Error Isolation

• If one request's constraint logic throws an error, it could affect the entire batch
• Need careful error boundaries so one bad request doesn't corrupt others

---

Step 7: Synthesis — The Full System View

┌─────────────────────────────────────────────────────────┐
│                    Caller Applications                   │
└──────────────────┬──────────────────┬───────────────────┘
                   │ gRPC             │ HTTP (OpenAI API)
┌──────────────────▼──────────────────▼───────────────────┐
│              JVM Serving System / MSS                    │
│         (routing, A/B testing, pre/post-processing)      │
└──────────────────────────┬──────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│                    NVIDIA Triton                          │
│              (model loading, batching, GPU scheduling)   │
│                                                          │
│  ┌─────────────────────────────────────────────────┐    │
│  │                    vLLM Engine                   │    │
│  │  ┌──────────────────────────────────────────┐   │    │
│  │  │  Custom Logits Processors (C++, batched) │   │    │
│  │  │  State machines per request              │   │    │
│  │  └──────────────────────────────────────────┘   │    │
│  └─────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────┐
│              Observability Layer                          │
│    Merged /metrics: Triton + vLLM (40+ metrics)         │
└─────────────────────────────────────────────────────────┘

---

Key Takeaways Summary

---

Self-Check Questions

1. Why does Netflix expose both gRPC and OpenAI-compatible HTTP APIs?
2. What is the GIL, and why did it cause problems in vLLM V0's logits processing?
3. When would you choose Versioned rollout over Red-Black?
4. Why is a silent API failure (like the response_format bug) more dangerous than a loud one?
5. What two metrics sources needed to be merged, and why didn't Triton's built-in bridge suffice?