Building a reliable LLM inference platform at Databricks to meet growing demands effectively.

Concept 1: What Is LLM Inference and Why Does Scale Make It Hard?

What is inference?

When you send a prompt to an AI model (like ChatGPT or Claude) and it generates a response, that process is called inference — the model is "inferring" an output from your input.

Why is scale hard?

• Millions of users send requests simultaneously
• Demand is spiky — it peaks during working hours and drops at night
• Each request has very different costs (a short question vs. a 100-page document analysis)

Key metrics to understand:

💡 Simple analogy: Think of a restaurant. At lunch rush, even a great kitchen slows down. LLM inference has the same problem, but the "dishes" vary wildly in complexity.

---

Concept 2: The Two Pillars — Availability vs. Latency

Availability

"Can the system process my request at all?"

• If a server crashes or is overloaded, your request fails entirely
• This is a binary problem: yes or no

Latency

"How fast does the system respond?"

• Even if a request succeeds, if it takes 30 seconds, it may be useless for real-time applications
• For AI agents (automated systems making decisions), slow responses can break entire workflows

Why both matter together

A system can be:

• ✅ Available but slow → bad for agents
• ✅ Fast but crashes under load → bad for everyone
• ✅ Both → what reliable inference requires

💡 Key insight: Latency problems can become availability problems. If a server gets too slow, it effectively becomes unavailable.

---

Concept 3: Why GPU Infrastructure Is Fundamentally Fragile

GPUs vs. CPUs — a reliability comparison

Why GPU failures are especially painful

All-to-all communication:

• Modern LLM serving splits work across many GPUs
• All GPUs must talk to each other constantly (called disaggregated prefill/decode)
• If one node goes down, it forces reconfiguration of multiple other nodes

Single-rack topology:

• The highest-speed GPU networks (like NVL72 systems) require all GPUs to be in one physical rack
• A failure in that rack = a wide blast radius (many things break at once)

Why standard fixes don't work here

💡 Analogy: Imagine a surgical team where every member must work in perfect sync. If one surgeon leaves mid-operation, the whole procedure is at risk — you can't just "add a backup surgeon in another room."

---

Concept 4: The Latency Challenge — Variable Request Costs

The core problem

Unlike a web server (where most requests take similar time), LLM requests vary enormously in cost:

• A short question: cheap and fast
• Summarizing a 500-page PDF: expensive and slow
• Generating code with images: very expensive

Why this is hard to predict

• You don't know how long the model will respond until it's done
• A request that looks short might trigger a very long answer
• This makes planning capacity extremely difficult

The throughput vs. latency tradeoff

More requests on one server → Higher throughput (cost efficient)
                           → But each request gets slower (higher latency)

💡 Analogy: A highway with more cars moves more total people, but each car moves slower. LLM servers face the same tradeoff.

---

Concept 5: Model Units — A Common Language for Request Cost

The problem it solves

How do you compare the "cost" of:

• A text-only request with 100 tokens?
• An image + text request with 1,000 tokens?
• A video analysis request?

You need a single unit of measurement.

What are Model Units?

A model unit is an abstraction that estimates how much of a server's capacity a request will consume.

The formula

Model Units = α × (input tokens) + β × (output tokens) + γ × (image tokens)

• α, β, γ are coefficients determined by automated benchmarking
• Different for each model and hardware type
• Can be adjusted for optimizations like prefix caching (reusing previously computed results)

Why this matters

💡 Analogy: Instead of measuring a restaurant's busyness by "number of customers," you measure it by "total cooking time needed." A table ordering 10 courses is very different from a table ordering coffee.

---

Concept 6: Smart Load Balancing with Dicer

What is load balancing?

When many servers exist, a load balancer decides which server handles each incoming request.

Traditional approach: Power of Two Choices (P2C)

• Pick 2 random servers
• Send the request to whichever has a shorter queue
• Works well for simple web servers

Why P2C fails for LLMs

• LLM requests take a long time (seconds to minutes)
• Server counts are lower than typical web systems
• Misrouting one expensive request can overload a server
• Queue length alone doesn't reflect true load

Databricks' solution: Dicer

Dicer routes based on model unit utilization instead of request count:

Traditional: "Server A has 10 requests, Server B has 15 → send to A"
Dicer:       "Server A is at 90% model unit capacity → send to B"

Bonus feature: Sticky routing (stateful sessions)

• A user's requests go to the same subset of servers
• This improves cache hit rates (the server already has context from previous requests)
• Especially important for coding agents that send many related requests

💡 Analogy: Instead of sending customers to whichever checkout line is shortest, you send them to the cashier who already knows their loyalty account — faster for everyone.

---

Concept 7: Autoscaling Based on Model Units

What is autoscaling?

Automatically adding or removing servers based on current demand.

The problem with naive autoscaling

Naive signal: "I have 100 pending requests → scale up"
Problem: 100 short requests ≠ 100 long requests

CPU and memory metrics are also uncorrelated with actual GPU utilization for LLMs.

Model-unit-based autoscaling

If model unit utilization > threshold → scale UP (add servers)
If model unit utilization < threshold → scale DOWN (remove servers)

The result

• Works for any model without manual tuning
• For bursty traffic: kept replica counts close to actual demand
• Saved over 80% in GPU costs compared to always running at peak capacity

💡 Analogy: A smart taxi dispatch system that counts "passenger-miles needed" rather than just "number of passengers" — a trip across town counts more than a trip around the block.

---

Concept 8: Detecting and Recovering from Silent Hangs

What is a silent hang?

A server stops responding but doesn't crash or throw an error. It just... freezes.

Common causes:

• Edge cases in structured output generation
• Multimodal inputs (images, video) triggering unhandled errors
• Multi-process architecture failures

Why silent hangs are dangerous

• The server looks "alive" to monitoring systems
• Requests pile up waiting for a response that never comes
• No error is surfaced → hard to detect

The solution: Black-box health checks

Periodically send a minimal test request to each server
If no real requests have completed recently AND the test fails → restart the server

This works regardless of which inference engine is being used.

The complication: Health checks under load

Problem: Under heavy load, health checks themselves time out → system thinks healthy servers are broken → kills them → cascading failure

Solution: Give health check requests the highest scheduling priority

• They always complete, even under heavy load
• False restarts dropped from several per week to zero
• Full detect → kill → recover cycle: under 5 minutes

💡 Analogy: A hospital's "code blue" alarm system. If the alarm itself gets stuck in a queue, you have a bigger problem. It must always have priority access.

---

Concept 9: CPU Bottlenecks in Multimodal Serving

The unexpected problem

When large batches of image requests arrived, error rates spiked — but the GPU was fine.

Root cause: Image processing is CPU-intensive, not just GPU-intensive:

• Decoding image files
• Resizing and normalizing pixels
• Preparing data for the vision encoder

What was happening

Image request arrives
→ CPU starts processing image (slow)
→ Event loop gets blocked
→ No other requests can be processed
→ Timeouts and errors spike

The discoveries from profiling

1. The image processing library being used was inefficient
2. OMP_NUM_THREADS (a setting controlling CPU thread usage) was misconfigured, causing CPU throttling
3. Moving work to separate threads/processes wasn't enough — the bottleneck was in the library itself

The fix

• Switched to Torchvision-based image processors (more efficient)
• Properly configured OMP_NUM_THREADS

The result

• Requests completed per second jumped more than 3x
• CPU throttling disappeared
• Same hardware, dramatically better performance

💡 Analogy: A factory assembly line where the bottleneck isn't the main machine (GPU) but the person unpacking boxes (CPU image processing). Speeding up the unpacking unlocks the whole line.

---

Summary: The Full Picture

User Request
     ↓
[Load Balancer - Dicer]
  Routes based on Model Unit utilization
  Sticky sessions for cache efficiency
     ↓
[Autoscaler]
  Adds/removes servers based on Model Unit utilization
  Saves 80%+ GPU costs vs. static provisioning
     ↓
[Inference Server]
  Health checks detect silent hangs
  Prioritized health checks prevent false restarts
  Optimized image processing prevents CPU bottlenecks
     ↓
Response

Core principles learned

---

🎯 The big takeaway: Reliable LLM inference at scale requires rethinking every assumption from traditional distributed systems — from how you measure load, to how you route requests, to how you detect failures — because LLM workloads are fundamentally different from web traffic.