Step-by-Step Teaching

Step 1: Understand the Problem First

Before any solution makes sense, you need to understand what token efficiency means and why it matters.

What is a "token"?

A token is a small unit of text (roughly 3-4 characters or 0.75 words). AI models charge based on tokens processed — both input (what you send) and output (what the model generates).

Why do agents spend so many tokens?

An AI agent doesn't just answer once. It works in multiple turns:

Turn 1: [System Prompt] + [Tool Definitions] + [User Message] → Model responds
Turn 2: [System Prompt] + [Tool Definitions] + [Turn 1 history] + [New message] → Model responds
Turn 3: [System Prompt] + [Tool Definitions] + [Turn 1+2 history] + [New message] → Model responds

Notice the problem: Every single turn re-sends the system prompt and tool definitions. As conversations grow longer, token costs compound rapidly.

The three types of token spend:

Key insight: The goal is to shift spending from uncached → cached, and eliminate unnecessary tokens entirely.

---

Step 2: Strategy #1 — Trim the System Prompt

What is a system prompt?

The system prompt is a set of instructions sent to the model before every conversation turn. It tells the agent how to behave, what rules to follow, and how to use tools.

The original problem:

Early AI models needed very explicit instructions:

• "DO NOT output binary data"
• "You MUST cite line numbers"
• "IMPORTANT: Do not use emojis"

These guardrails were necessary because models were less capable.

The solution:

As models improved, they learned these behaviors naturally. The explicit instructions became redundant dead weight.

Before: Long lists of "DO NOT", "You MUST", "IMPORTANT" rules
After:  Simply define what a tool does — models comply on their own

Result: ~66% reduction in system prompt size

Why this matters conceptually:

Think of it like employee training. A new hire needs a 50-page manual. A senior expert needs a one-page brief. The instructions didn't get less important — the recipient got smarter.

Principle learned: Regularly audit static instructions. What was necessary for less capable systems may be unnecessary overhead for more capable ones.

---

Step 3: Strategy #2 — Load Tools Only When Needed

What are tool definitions?

When an agent can use tools (like searching the web, reading files, running shell commands), each tool's full description must be included in the request so the model knows the tool exists and how to use it.

The problem:

Cursor added many powerful tools over time. But here's the critical insight:

Most tools are needed in fewer than 20% of conversations

Sending all tool definitions every turn is like carrying every tool in your toolbox to every job site — even when you only need a hammer.

The solution — Static vs. Dynamic context:

STATIC CONTEXT (always included):
├── High-frequency tools (read, search, edit, shell)
├── Tools models hallucinate without seeing (ask_question)
└── Product-critical tools (create_plan for Plan Mode)

DYNAMIC CONTEXT (loaded only when needed):
└── All other tools → loaded on-demand

How they decided what stays static:

They ran A/B tests measuring:

• Token usage
• Cost
• Latency
• Tool-call errors
• Overall agent usage

This is important — they didn't just guess. They measured quality impact alongside efficiency gains.

Result: 60% reduction in static-context description tokens

Principle learned: Separate "always needed" from "sometimes needed." Load resources on demand rather than preemptively.

---

Step 4: Strategy #3 — Improve Cache Reuse

What is prompt caching?

When you send the same text to a model repeatedly, the provider can cache (save) the processed version and reuse it instead of reprocessing from scratch. Cached tokens cost significantly less.

The problem with caching:

For caching to work, the beginning of the request must be identical across turns. But if variable content (like user-specific settings) was mixed into the stable content (like tool definitions), the cache would break.

BEFORE (cache breaks easily):
[System Prompt + Variable User Settings + Tool Definitions + Conversation]
         ↑ This changes → everything after it can't be cached

The solution — Explicit cache breakpoints:

AFTER (cache preserved):
[Stable: System Prompt] ← BREAKPOINT
[Stable: Tool Definitions] ← BREAKPOINT  
[Variable: User settings, environment info] ← "phantom user message"
[Growing: Conversation history]

By placing explicit breakpoints after stable content, later turns can reuse the unchanged prefix even as the conversation grows.

The "phantom user message" concept:

Variable, request-specific content (skills, subagent info, environment details) was moved past the cache boundaries into a special container. This keeps the cacheable prefix clean and stable.

Result: 20% reduction in cold cache misses

Principle learned: Structure your data so that stable content comes first and variable content comes last. This maximizes the reusable prefix for caching.

---

Step 5: Strategy #4 — Compress File Reads

The problem:

When an agent reads code files, it traditionally numbered every single line because:

• Models can't count lines reliably on their own
• Users need to reference specific line numbers

But line numbers add tokens:

1  def hello():
2      print("world")
3      return True

When reading tens of thousands of lines per session, this adds up significantly.

The solution:

Number only every 10th line instead of every line:

1  def hello():
   def goodbye():
   def maybe():
10 def another():
   def function():

Models can still cite code accurately (interpolating between numbered lines), but token overhead drops substantially.

Result: 1.6% reduction in cache-read tokens with no quality loss

Principle learned: Small per-unit savings multiply dramatically at scale. Even a few tokens per line × millions of lines = meaningful cost reduction.

---

Step 6: Strategy #5 — Use Subagents Strategically

What is a subagent?

A subagent is a separate AI agent spawned by the main (parent) agent to handle a specific subtask. The key efficiency benefit:

Parent Agent: [Full conversation history — 50,000 tokens]
     ↓ spawns
Subagent: [Fresh context — starts at 0 tokens]
     ↓ completes task, returns summary
Parent Agent: [Receives compact result, not full subagent history]

The trade-off — coordination tax:

Subagents don't share context with each other, which means:

• They might duplicate work already done
• They might pursue tasks that are no longer necessary

This is the fundamental tension: isolation saves tokens, but coordination costs tokens

Two fixes applied:

Fix 1: Remove over-eager subagent instructions Earlier prompts strongly encouraged using subagents for codebase exploration. But models had learned this pattern naturally through training. The extra instructions were causing too many subagents to be spawned unnecessarily.

Before: Explicit instructions → models over-use subagents
After:  Remove instructions → models use subagents more appropriately

Fix 2: Tighten model selection for subagents Cursor can spawn subagents using different (cheaper) models. But agents were choosing different models too freely. Now they only switch models when:

• The user explicitly requests it, OR
• The system harness directs it

Result: More balanced subagent usage without unnecessary coordination overhead

Principle learned: Context isolation between agents is powerful but has coordination costs. Use subagents when the task is truly independent; avoid them when shared context is needed.

---

Step 7: Putting It All Together

Here's how all five strategies interact as a unified system:

EVERY AGENT TURN:
┌─────────────────────────────────────────────┐
│ STATIC (cached, trimmed)                    │
│  • Lean system prompt (-66%)                │
│  • Only high-frequency tools (-60%)         │
│  ← CACHE BREAKPOINT →                       │
│ VARIABLE (past cache boundary)              │
│  • User settings, environment               │
│  ← CACHE BREAKPOINT →                       │
│ DYNAMIC (loaded on demand)                  │
│  • Rare tools (loaded only when needed)     │
│ GROWING (conversation history)              │
│  • File reads with sparse line numbers      │
│  • Subagent results (not full histories)    │
└─────────────────────────────────────────────┘

Combined result: 7% reduction in total token costs without quality degradation

---

Summary Table

---

Core Principles to Remember

1. Audit regularly — What was necessary for less capable systems may be dead weight now
2. Separate static from dynamic — Load resources only when actually needed
3. Structure for caching — Stable content first, variable content last
4. Small savings multiply — Tiny per-unit reductions matter enormously at scale
5. Measure quality alongside efficiency — Never optimize cost without tracking quality impact
6. Isolation vs. coordination — Context boundaries between agents save tokens but introduce coordination costs; balance deliberately