Databricks enhances open-source LLMs with prompt caching to boost speed and reduce costs.

Concept 1: The Core Problem - Repeated Prompts

What's happening?

When you use an LLM in production, many requests share identical text at the beginning (called a "prefix").

Request 1: [System Prompt - 2000 tokens] + [User Question A]
Request 2: [System Prompt - 2000 tokens] + [User Question B]
Request 3: [System Prompt - 2000 tokens] + [User Question C]

Why is this wasteful?

Every single request forces the model to reprocess that identical system prompt from scratch.

Simple Analogy: Imagine re-reading an entire textbook every time someone asks you a question about it, instead of remembering what you already read.

---

Concept 2: What is a System Prompt / Instruction Prompt?

Definition

A system prompt is a fixed set of instructions given to the LLM before the actual user question.

┌─────────────────────────────────────────┐
│ SYSTEM PROMPT (fixed, repeated)         │
│ "You are a helpful assistant. You work  │
│  in the medical domain. Always cite     │
│  sources. Never give diagnoses..."      │
│ [potentially thousands of tokens]       │
├─────────────────────────────────────────┤
│ USER QUESTION (changes each request)    │
│ "What are symptoms of diabetes?"        │
└─────────────────────────────────────────┘

Key insight from the article

• Frontier models like Claude use system prompts thousands of tokens long
• Enterprise use cases often need domain-specific system prompts
• These prompts are shared across thousands of requests

---

Concept 3: KV Cache - The Technical Foundation

What is a KV Cache?

When an LLM processes text, it generates Key-Value (KV) pairs — intermediate mathematical representations for each token.

Text Input → Transformer Layers → KV Pairs (stored in memory)
                                        ↓
                              Used to generate next token

Normal flow (WITHOUT caching):

Request 1: Process [2000 token prompt] → Generate KV pairs → Answer
           ❌ KV pairs discarded

Request 2: Process [2000 token prompt] → Generate KV pairs → Answer
           ❌ KV pairs discarded again

Request 3: Process [2000 token prompt] → Generate KV pairs → Answer
           ❌ Discarded again...

With KV Cache:

Request 1: Process [2000 token prompt] → Generate KV pairs → Answer
           ✅ KV pairs SAVED in memory

Request 2: Load saved KV pairs → Process only new tokens → Answer
           ✅ Much faster!

Request 3: Load saved KV pairs → Process only new tokens → Answer
           ✅ Fast again!

Simple Analogy: Instead of solving the same math equation from scratch every time, you save your work and only solve the new part.

---

Concept 4: Prompt Caching - The Solution

Definition

Prompt caching is the technique of storing the KV cache of a repeated prompt prefix so it can be reused across multiple requests.

How it works visually:

                    ┌──────────────────────┐
                    │   PROMPT CACHE       │
                    │   (Memory Storage)   │
                    │                      │
                    │  [KV pairs for       │
                    │   shared prompt]     │
                    └──────────┬───────────┘
                               │ reuse
              ┌────────────────┼────────────────┐
              ↓                ↓                ↓
         Request 1        Request 2        Request 3
         [New tokens]     [New tokens]     [New tokens]
         only!            only!            only!

Benefits delivered:

---

Concept 5: Amortization of Compute Cost

What does "amortized cost" mean here?

The article says: "the compute cost of that shared prompt [is] amortized across all those queries"

Breaking it down:

WITHOUT caching:
Cost per request = Full prompt cost + Question cost
1000 requests × (2000 token cost + 50 token cost) = VERY HIGH

WITH caching:
First request  = Full prompt cost + Question cost  ← pay once
Next 999 requests = Only question cost             ← pay small amount

Total cost spread (amortized) across all 1000 requests = MUCH LOWER

Simple Analogy: Like buying a monthly bus pass. You pay once and ride many times, making each ride cheaper than buying individual tickets.

---

Concept 6: Quality vs. Throughput Trade-off (Now Resolved)

The old dilemma:

Longer system prompt = Better quality (more context, instructions)
                     = BUT slower throughput (more tokens to process)

Shorter system prompt = Faster throughput
                      = BUT lower quality

How prompt caching resolves this:

With Prompt Caching:
Longer system prompt = Better quality ✅
                     = NO throughput penalty ✅ (cached!)

You get BOTH quality AND speed!

The article's research finding:

• Automated prompt optimization + open-source models
• Can surpass frontier model quality for enterprise tasks
• Prompt caching makes this economically viable

---

Concept 7: Open-Source vs. Proprietary Models

Two categories of LLMs:

┌─────────────────────────┐    ┌─────────────────────────┐
│   PROPRIETARY MODELS    │    │   OPEN-SOURCE MODELS    │
│   (Closed weights)      │    │   (Open weights)        │
├─────────────────────────┤    ├─────────────────────────┤
│ • GPT (OpenAI)          │    │ • Llama (Meta)          │
│ • Gemini (Google)       │    │ • Mistral               │
│ • Claude (Anthropic)    │    │ • DBRX (Databricks)     │
└─────────────────────────┘    └─────────────────────────┘
        ↓                               ↓
Databricks had prompt              NOW ALSO has
caching ALREADY                    prompt caching! ✅

Why this matters:

• Many enterprises prefer open-source for control, privacy, cost
• Previously, they missed out on prompt caching benefits
• Databricks has now closed this gap

---

Concept 8: Security Design of the Cache

Why security matters for caching:

Company A's system prompt → cached
                                    ← Could Company B access it? ❌ NO!
Company B's system prompt → cached

Databricks' security approach:

┌─────────────────────────────────────────────┐
│           SECURITY PRINCIPLES               │
├─────────────────────────────────────────────┤
│ 1. ISOLATED    → Each customer's cache is   │
│                  completely separate        │
│                                             │
│ 2. VOLATILE    → Only lives in RAM          │
│                  (disappears when cleared)  │
│                                             │
│ 3. NOT PERSISTED → Never written to disk    │
│                    or long-term storage     │
└─────────────────────────────────────────────┘

Simple Analogy: Like a whiteboard in a private room — only you can see it, and it gets erased when you leave.

---

Concept 9: Implicit vs. Explicit Caching

Two approaches to caching:

Explicit Caching (NOT what Databricks does):

# Developer must manually configure
response = llm.call(
    prompt="...",
    cache_prefix=True,        # ← must remember to add this
    cache_id="my_cache_123"   # ← must manage cache IDs
)

Implicit Caching (Databricks approach):

# Developer just makes normal API call
response = llm.call(
    prompt="..."
    # That's it! Caching happens automatically ✅
)

Why implicit is better:

---

Concept 10: Where Prompt Caching Applies

Workload types supported:

┌─────────────────────────────────────────────────────┐
│              DATABRICKS WORKLOADS                   │
├─────────────────┬───────────────────────────────────┤
│ Batch Inference │ Process large volumes of requests │
│                 │ offline (e.g., document analysis) │
├─────────────────┼───────────────────────────────────┤
│ Pay-per-token   │ Real-time API calls, pay for      │
│                 │ what you use                      │
├─────────────────┼───────────────────────────────────┤
│ Provisioned     │ Reserved capacity for consistent  │
│ Throughput      │ high-volume workloads             │
└─────────────────┴───────────────────────────────────┘

Higher-level services that also benefit:

Prompt Caching
      ↓
Powers these services automatically:
├── Agent Bricks  (AI agents)
├── Genie         (data analytics AI)
└── AI Functions  (SQL-integrated AI)

---

Concept 11: Real-World Performance Results

What Databricks observed in production:

The article mentions measurable gains in large-scale batch inference pipelines after rolling out to GPT-OSS models:

BEFORE prompt caching:
[████████████████████] Full prompt processed every request
High latency | High cost | Lower throughput

AFTER prompt caching:
[████] Only new tokens processed
Lower latency ✅ | Lower cost ✅ | Higher throughput ✅

Use cases that benefit most:

High benefit scenarios:
├── 🤖 Real-time chat (same system prompt, many users)
├── 📄 Batch document processing (same instructions, many docs)
└── 🔧 AI agents (long context windows, repeated tool descriptions)

---

Summary: The Complete Picture

THE JOURNEY OF A PROMPT WITH CACHING:

1. REQUEST ARRIVES
   "System: You are an expert... [2000 tokens]
    User: What is X?"

2. SYSTEM CHECKS CACHE
   "Have I seen this prefix before?"
        ├── YES → Load KV cache, skip to user question ⚡
        └── NO  → Process full prompt, SAVE to cache 💾

3. GENERATE RESPONSE
   Only process the new/unique tokens

4. RESULT
   ✅ Faster response
   ✅ Lower compute cost
   ✅ Higher throughput
   ✅ Better quality (can use longer prompts)
   ✅ Secure (isolated, volatile, not persisted)
   ✅ Zero configuration needed

---

Key Takeaways