Bridging the intent-execution gap enhances AI agent performance across diverse benchmarks.

Concept 1: AI Agents Are Systems, Not Just Models

What it means

When people think about AI performance, they often focus only on the AI model itself (like GPT or Claude). But in reality, a working AI agent is a combination of two things:

1. The LLM (Large Language Model) — the "brain" that reasons and decides
2. The Harness — the software "operating system" around the model that:
   • Connects the model to tools (like a terminal or file editor)
   • Manages the cycle of: think → act → observe → repeat

Analogy

Think of it like a surgeon and a hospital:

• The surgeon = the LLM (the intelligent decision-maker)
• The hospital infrastructure = the harness (tools, rooms, nurses, equipment)

Even the best surgeon performs poorly in a broken hospital.

Key takeaway

As models get smarter, the harness becomes the bottleneck, not the model.

---

Concept 2: The Intent-Execution Gap

What it means

This is the central problem of the article. It describes the mismatch between:

• What the model intends to do → and what the harness actually executes
• What the harness actually did → and what the model understands happened

Simple Example

The model wants to fix one specific function in a codebase. The harness finds the same function name in 5 places and edits all of them.

The model's intent was clear. The execution was wrong.

It's Bidirectional

Key takeaway

Closing this two-way gap — without any task-specific tuning — is enough to achieve state-of-the-art performance.

---

Concept 3: Tool-Interface Failures (How Editing Goes Wrong)

The article focuses on a common agent task: editing code. It identifies three specific failure modes in the file-editing tool.

---

Failure Mode 1: Ambiguous Matches (Multiple Locations)

What happens

The model says: "Replace this text with that text." But the same text appears in 5 different places in the codebase.

The harness doesn't know which one the model meant → it either:

• Replaces all of them (dangerous), or
• Picks one arbitrarily (unreliable)

The fix

The harness should stop and ask the model to provide more context so the target location becomes unique.

Key insight

A small implementation detail (asking for clarification) sharply improves faithfulness between intent and execution.

---

Failure Mode 2: Partial-Line Matching (Weak Anchors)

What happens

The model provides only a short fragment of text to replace (e.g., a partial line).

That fragment might appear:

• Inside comments
• Inside string literals
• In unrelated code paths

Even if the fragment is unique, replacing a partial line can produce syntactically valid but semantically broken code.

The fix

Require stronger anchors — full lines, surrounding context, or line-number-aware matching.

Key insight

The harness should not guess when edit requests are underspecified.

---

Failure Mode 3: No Feedback After Editing (Fire-and-Forget)

What happens

After a successful edit, the harness just says: "Edit succeeded."

The model has no way to verify:

• Did the edit land in the right place?
• Were any unintended lines changed?
• Is a follow-up edit needed?

The fix

After every edit, provide the model with a diff file — a structured summary showing:

• What was added ✅
• What was deleted ❌
• What stayed the same

Key insight

This converts editing from a fire-and-forget action into an observable state transition.

---

Why Not Just Rely on the Diff to Catch All Mistakes?

A natural question: "If the diff shows mistakes, why fix Failure Modes 1 and 2 separately?"

The answer is about when the correction happens:

If you only use the diff:

• The wrong edit is already applied
• The model must spend extra reasoning effort to undo or repair it
• This fills the context window with irrelevant information, degrading future decisions

Prevention is better than correction — but both are necessary.

---

Concept 4: Managing Tool Output Length

What it means

The bash terminal tool is very powerful but can produce extremely long outputs.

Simply cutting off (truncating) the output at the end is bad because:

• The end of output often contains critical information (job status, success/failure messages)

The fix

Condense the middle of long outputs, keeping:

• A limited number of lines at the beginning
• A limited number of lines at the end

Key insight

Smart output management preserves the most diagnostically useful information for the model.

---

Concept 5: Balancing Reasoning and Tool Use (The Reasoning Nudge)

The tension

Agents need to think (chain-of-thought reasoning) AND act (call tools). But there's a fundamental tension:

The solution: The Reasoning Nudge

Encourage the model to reason just enough to decide the next action, then prioritize tool interaction over further internal reasoning.

The idea: validate hypotheses through tool outputs, not more thinking.

It's Model-Specific

Different model families respond differently to nudges:

Key insight

There is no single golden prompt for balancing reasoning and action across all models.

---

Concept 6: Model-Specific Tool Preferences

What it means

Even when tools work identically, different models prefer to use them differently — as a result of how they were trained.

Examples from the article

Key insight

Agent performance depends not just on what tools are available, but on how naturally those tools align with the model's learned behaviors.

A good harness meets the model where it is rather than forcing all models into the same interface.

---

Concept 7: Benchmaxing — A Warning About Benchmark Scores

What it means

Benchmaxing = reporting inflated benchmark numbers that don't reflect true model/harness capability.

Why it happens

Benchmark scores are influenced by infrastructure parameters that have nothing to do with intelligence:

• Environment interaction timeouts
• Infrastructure stability
• Network bandwidth
• Number of concurrent tasks on a single machine
• API reliability (tokens per minute, request limits)

Real example from the article

In Terminal-Bench-2, if multiple projects share one machine:

• Network bandwidth is split among all projects
• Downloading dependencies (like PyTorch) takes longer
• Less time remains for actual problem-solving
• More timeouts → lower scores

Key insight

A higher benchmark score might just mean better infrastructure, not a smarter agent or harness.

---

Concept 8: Constrained vs. Unconstrained Evaluation

What it means

Some benchmarks (like Terminal-Bench-2) impose resource limits:

• Memory limits
• Time limits per task
• CPU limits

These limits can artificially suppress performance.

The finding

When time and memory constraints are removed (unconstrained setting), performance improves by 5–10% on average.

Some specific tasks (like make-doom-for-mips, torch-pipeline-parallelism) consistently time out under constraints but succeed without them.

Key insight

Constrained benchmarks measure a mix of agent capability AND infrastructure efficiency — these should be understood separately.

---

Concept 9: Invariant Design Principles vs. Model-Specific Optimizations

The core problem with optimization

Many agent improvements are brittle:

• They work for one model version
• They break or regress with the next model version
• They overfit to current model behavior

The better approach: Find Invariants

Invariants = design principles that remain effective across:

• Different model versions
• Different benchmarks
• Different environments

What the article identifies as invariants

What remains model-specific

• Exact reasoning nudge phrasing
• Tool interface format preferences
• Line number display
• Batch vs. sequential command execution

Key insight

Build on invariant foundations, then apply minimal, targeted adaptations per model family.

---

Concept 10: The Simple Strands Agent (SSA)

What it is

SSA is a lightweight, open-source, single-agent harness that implements all the principles above.

Its goals

1. Close the gap between documented performance and real-world open-source performance
2. Achieve consistent gains across multiple models and benchmarks
3. Use minimal but non-zero model-specific adaptations within a shared framework

What "minimal adaptations" means

• The core harness is identical across all models
• Only prompts and tool specifications differ slightly per model family
• The goal is not separate agents per model, but orthogonal adaptations that let each model express its strengths

Key insight

SSA demonstrates that principled harness design — not model-specific tuning — is the primary driver of state-of-the-art performance.

---

Summary: The Full Picture

┌─────────────────────────────────────────────────────┐
│                    AI AGENT SYSTEM                  │
│                                                     │
│   ┌──────────┐         ┌──────────────────────┐    │
│   │   LLM    │◄───────►│       HARNESS        │    │
│   │ (Brain)  │         │ (Operating System)   │    │
│   └──────────┘         └──────────────────────┘    │
│                                                     │
│   The GAP between them = Intent-Execution Gap       │
└─────────────────────────────────────────────────────┘

To close the gap:
✅ Prevent ambiguous edits (before execution)
✅ Require strong text anchors (before execution)  
✅ Provide diff feedback (after execution)
✅ Manage output length intelligently
✅ Balance reasoning with tool use (reasoning nudge)
✅ Adapt tool interfaces to model preferences
✅ Build on invariant principles, not brittle optimizations

The central message of the article is:

Great AI agent performance comes from great systems design, not just great models.