How to Evaluate LLM Agents Beyond Task Success

Learning Outcome Definition

After studying this material, you should be able to:

1. Explain why single-metric evaluation (task success rate) is insufficient for assessing LLM agent quality
2. Identify and describe the eight metrics across three evaluation layers
3. Interpret metric gap patterns to diagnose specific behavioral failure modes
4. Apply the tiered evaluation framework to compare agent configurations
5. Distinguish between structurally different agent behaviors that appear identical under simple accuracy scoring

---

Step 1: The Core Problem — Why Task Success Alone Fails

The Fundamental Issue

Imagine two students both score 80% on a test. Does that mean they learned the same way? Not necessarily — one might have genuinely understood the material, while the other guessed strategically.

The same problem applies to LLM agents.

Agent A:  Accuracy = 65%
Agent B:  Accuracy = 65%

Are they equivalent? ❌ NOT NECESSARILY

What Hides Behind Identical Scores

Two agents with identical end-to-end accuracy can exhibit completely different production behaviors:

The Key Insight

Two configurations can achieve identical end-to-end accuracy while exhibiting structurally different production behaviors.

This means ranking agents by accuracy alone systematically misranks them for real-world deployment.

---

Step 2: The Solution — A Three-Layer Evaluation Framework

The protocol decomposes agent quality into three layers, each capturing a different dimension of behavior.

┌─────────────────────────────────────────────────────┐
│                  AGENT EVALUATION                    │
├─────────────────┬──────────────────┬────────────────┤
│   ANSWER LAYER  │ TRAJECTORY LAYER │ EXECUTION LAYER│
│                 │                  │                │
│ What did the    │ How did the agent│ How reliably   │
│ agent say?      │ plan its steps?  │ did it run?    │
└─────────────────┴──────────────────┴────────────────┘

Think of it like evaluating a surgeon:

• Answer layer = Did the patient recover? (outcome)
• Trajectory layer = Did they follow correct procedure? (process)
• Execution layer = Were there complications? (reliability)

---

Step 3: The Eight Metrics — One by One

Layer 1: Answer Metrics (3 metrics)

These measure the quality of the final output.

Metric 1: Accuracy

Simple binary or percentage score
Did the agent get the right answer?

Example:
  Expected: "Paris"
  Agent A:  "Paris"    → Accuracy = 1.0 ✓
  Agent B:  "Lyon"     → Accuracy = 0.0 ✗

Metric 2: Semantic F1

Measures overlap of meaning, not exact words
Combines Precision and Recall of semantic content

Example:
  Expected: "The revenue increased by 15% in Q3 2024"
  Agent:    "Q3 2024 saw a 15% revenue growth"
  
  Exact match: FAIL ✗
  Semantic F1: HIGH ✓  (same meaning captured)

Metric 3: Cosine Similarity

Measures directional similarity in embedding space
Range: 0.0 (unrelated) → 1.0 (identical meaning)

Useful for catching:
  - Paraphrased correct answers
  - Near-miss responses
  - Formatting-only differences

Why all three? Each catches different failure types:

Scenario: Agent gives correct info but wrong format
  Accuracy:        LOW  (exact match fails)
  Semantic F1:     HIGH (content is correct)
  Cosine Sim:      HIGH (meaning preserved)
  
Diagnosis: FORMATTING FAILURE, not reasoning failure

---

Layer 2: Trajectory Metrics (4 metrics)

These measure how the agent planned and executed its reasoning steps — the sequence of tool calls it made.

Key concept: A "trajectory" is the ordered sequence of tool calls an agent makes to reach its answer.

Example trajectory for "What is the CEO's salary at Company X?":

Step 1: search_company(name="Company X")
Step 2: get_executive_info(company_id=123, role="CEO")  
Step 3: get_compensation(executive_id=456)
→ Answer: "$2.3M"

Metric 4: Set Match

Did the agent use the RIGHT TOOLS? (order doesn't matter)

Expected tools: {search_company, get_executive_info, get_compensation}
Agent tools:    {search_company, get_executive_info, get_compensation}

Set Match = 1.0 ✓

Agent B tools:  {web_search, get_compensation}
Set Match = 0.5 ✗ (wrong tools selected)

Metric 5: Order Match

Did the agent use tools in the RIGHT SEQUENCE?

Expected: [search → get_executive → get_compensation]
Agent A:  [search → get_executive → get_compensation]  → Order Match = 1.0 ✓
Agent B:  [get_compensation → search → get_executive]  → Order Match = LOW ✗

Metric 6: Args Match

Did the agent pass the RIGHT ARGUMENTS to tools?

Expected: get_executive_info(company_id=123, role="CEO")
Agent A:  get_executive_info(company_id=123, role="CEO")  → Args Match = 1.0 ✓
Agent B:  get_executive_info(company_id=999, role="CEO")  → Args Match = LOW ✗
                                          ↑
                              HALLUCINATED company_id!

Args Match is critical — it catches silent argument hallucination, where the agent calls the right tool but with invented parameters.

Metric 7: Tool Coverage

Did the agent use ALL necessary tools?
(Penalizes missing tools even if some were correct)

Expected tools: 3 tools needed
Agent used:     2 tools (skipped one)

Tool Coverage = 2/3 = 0.67

High Set Match + Low Tool Coverage = agent used correct tools BUT missed some

---

Layer 3: Execution Metrics (1 metric)

Metric 8: Average Failures per Trajectory (Avg Failures/Traj)

How many tool call failures occurred on average?

Agent A: 0 failures across 50 queries    → Avg Failures/Traj = 0.0
Agent B: 75 failures across 50 queries   → Avg Failures/Traj = 1.5

Includes:
  - API errors
  - Timeout retries  
  - Invalid argument rejections
  - Tool loop terminations

The declining pattern matters:

Query 1:  2 failures (agent retried, eventually succeeded)
Query 25: 1 failure  (agent learning/adapting)
Query 50: 0 failures (agent stabilized)

Non-zero BUT DECLINING Avg Failures/Traj = "Resilient" behavior signature

---

Step 4: Reading Metric Patterns — Diagnosing Failure Modes

This is where the framework's diagnostic power emerges. You don't read metrics individually — you read gap patterns across layers.

The Five Behavioral Signatures

Signature 1: Resilient Self-Correction

Set Match:    HIGH  ✓
Order Match:  LOW   ✗
Avg Failures: Non-zero but DECLINING

Interpretation:
  Agent uses correct tools but reorders them adaptively
  Encounters errors but recovers
  Gets right answer through flexible, self-correcting path
  
  This is GOOD behavior — adaptive, robust

Signature 2: Compliant-but-Fragile Planning

Accuracy:     HIGH  ✓
Set Match:    HIGH  ✓
Order Match:  HIGH  ✓
Args Match:   LOW   ✗

Interpretation:
  Agent follows the right plan on the surface
  But passes wrong arguments to tools
  Gets lucky with correct answers despite bad tool calls
  
  This is DANGEROUS — works in testing, fails in production

Signature 3: Formatting-Only Failures

Accuracy:     LOW   ✗
Semantic F1:  HIGH  ✓
Cosine Sim:   HIGH  ✓

Interpretation:
  Agent has the RIGHT KNOWLEDGE
  But outputs it in wrong format
  
  Fix: Output formatting, not reasoning

Signature 4: Tool Misrouting

Set Match:      LOW  ✗
Tool Coverage:  LOW  ✗
Accuracy:       LOW  ✗

Interpretation:
  Agent selects wrong tools entirely
  Reasoning about WHICH tool to use is broken
  
  Fix: Tool selection logic, not answer generation

Signature 5: Execution Environment Failures

Accuracy:     LOW   ✗
Set Match:    HIGH  ✓
Args Match:   HIGH  ✓
Avg Failures: HIGH  ✗

Interpretation:
  Agent plans correctly and argues correctly
  But tools keep failing (API down, timeouts, etc.)
  
  Fix: Infrastructure, not agent logic

Diagnostic Decision Tree

Start: Agent performing poorly
         │
         ▼
    Check Answer Layer
    ┌────────────────────────────────┐
    │ Semantic F1 high, Accuracy low?│──YES──→ FORMATTING FAILURE
    └────────────────────────────────┘
         │NO
         ▼
    Check Trajectory Layer
    ┌──────────────────────┐
    │ Set Match low?       │──YES──→ TOOL MISROUTING
    └──────────────────────┘
         │NO
    ┌──────────────────────┐
    │ Args Match low?      │──YES──→ ARGUMENT HALLUCINATION
    └──────────────────────┘
         │NO
         ▼
    Check Execution Layer
    ┌──────────────────────┐
    │ Avg Failures high?   │──YES──→ ENVIRONMENT FAILURE
    └──────────────────────┘
         │NO
         ▼
    Check Order Match
    ┌──────────────────────────────────────┐
    │ Set Match high, Order Match low,     │
    │ Failures declining?                  │──YES──→ RESILIENT (GOOD!)
    └──────────────────────────────────────┘

---

Step 5: The Five Agent Configurations Tested

The paper tests this framework on five agent types:

┌─────────────────────────────────────────────────────────────┐
│                    AGENT CONFIGURATIONS                      │
├──────────────────────┬──────────────────────────────────────┤
│ Configuration        │ Description                          │
├──────────────────────┼──────────────────────────────────────┤
│ CoT (Zero-shot)      │ Chain-of-Thought, no optimization    │
│ CoT (DSPy-Simba)     │ CoT with automated prompt tuning     │
│ ReAct (Zero-shot)    │ Reason+Act loop, no optimization     │
│ ReAct (DSPy-Simba)   │ ReAct with automated prompt tuning   │
│ CodeAgent            │ Generates and executes code          │
└──────────────────────┴──────────────────────────────────────┘

Key Concepts

Chain-of-Thought (CoT): Agent reasons step-by-step before answering

Think: "First I need X, then Y, then Z"
→ Execute plan
→ Answer

ReAct: Agent interleaves reasoning and acting

Reason → Act → Observe → Reason → Act → Observe → Answer

DSPy-Simba optimization: Automated system that tunes agent prompts using examples

---

Step 6: Key Findings — What the Eight Metrics Revealed

Finding 1: Optimized ReAct Wins, But Why?

ToolHop Benchmark:
  ReAct (DSPy-Simba): 65% accuracy  ← BEST

Production Set (50 real queries):
  ReAct (DSPy-Simba): 86% accuracy  ← BEST

But accuracy alone doesn't explain why it wins. The eight metrics show:

ReAct (DSPy-Simba) behavioral signature:
  Set Match:    HIGH   → Uses right tools
  Order Match:  MEDIUM → Adapts sequence when needed
  Args Match:   HIGH   → Correct arguments
  Avg Failures: Low but non-zero, declining → Self-corrects
  
= RESILIENT signature

Finding 2: DSPy Optimization Has a "Glass Ceiling" for CoT

CoT (Zero-shot) vs CoT (DSPy-Simba):

  Accuracy:     Improves ✓
  Semantic F1:  Improves ✓
  Args Match:   Improves ✓  ← DSPy fixes syntax
  Order Match:  No change ✗ ← DSPy can't fix planning
  
Interpretation:
  DSPy optimization patches FORMATTING problems in CoT
  But cannot unlock ADAPTIVE REASONING
  
  CoT is a "static planner" — it commits to a plan upfront
  and cannot adapt when tools fail or return unexpected results

This is the "glass ceiling" — optimization improves surface compliance but hits a structural limit.

Finding 3: The Resilient Signature Transfers

Synthetic benchmark (ToolHop):
  ReAct shows: High Set Match, variable Order Match, declining failures
  
Real production queries (50 enterprise queries):
  ReAct shows: SAME PATTERN
  
Conclusion: The behavioral signature is STABLE across contexts
            Benchmark findings predict production behavior

This validates the protocol — metrics measured on benchmarks predict real deployment behavior.

---

Step 7: Implementation Concepts

The Stateful Metric Callable

The protocol is implemented as a stateful metric callable — a measurement system that:

Traditional evaluation:
  Run agent → Get answer → Score answer → Done

Stateful metric callable:
  Run agent → Capture full trace → Store trace → Score later
                                                      ↑
                                          Can re-score with NEW metrics
                                          against OLD logged traffic

Why this matters: If you define a new metric tomorrow, you can apply it to yesterday's traffic without re-running the agent.

The Trace-Extraction Harness

Agent execution produces:
  ┌─────────────────────────────────────┐
  │ TRACE                               │
  │  - Tool call 1: name, args, result  │
  │  - Tool call 2: name, args, result  │
  │  - Failures: timestamps, types      │
  │  - Final answer: text               │
  └─────────────────────────────────────┘
         │
         ▼
  Trace-extraction harness
  (decouples measurement from deployment)
         │
         ▼
  Eight-metric scorer

The harness decouples measurement from deployment — you can score traces from any agent, any deployment, retroactively.

---

Step 8: Putting It All Together — The Complete Framework

┌─────────────────────────────────────────────────────────────────┐
│              EIGHT-METRIC EVALUATION PROTOCOL                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ANSWER LAYER          TRAJECTORY LAYER       EXECUTION LAYER   │
│  ┌─────────────┐       ┌─────────────────┐    ┌─────────────┐  │
│  │ 1. Accuracy │       │ 4. Set Match    │    │ 8. Avg      │  │
│  │ 2. Sem. F1  │       │ 5. Order Match  │    │    Failures │  │
│  │ 3. Cos. Sim │       │ 6. Args Match   │    │    /Traj    │  │
│  └─────────────┘       │ 7. Tool Coverage│    └─────────────┘  │
│                         └─────────────────┘                     │
│                                                                  │
│  READ JOINTLY → Identify gap patterns → Diagnose failure mode   │
│                                                                  │
│  Failure Modes:                                                  │
│  • Resilient Self-Correction (GOOD)                             │
│  • Compliant-but-Fragile Planning (DANGEROUS)                   │
│  • Formatting-Only Failures (EASY FIX)                          │
│  • Tool Misrouting (TOOL SELECTION BROKEN)                      │
│  • Execution Environment Failures (INFRASTRUCTURE)              │
└─────────────────────────────────────────────────────────────────┘

---

Summary: Key Takeaways

---

Self-Assessment Questions

1. Two agents both achieve 70% accuracy. Agent A has high Semantic F1 but low Accuracy. Agent B has matching Accuracy and Semantic F1. What does this tell you about each agent?

2. An agent shows: Set Match = 0.9, Args Match = 0.3, Accuracy = 0.7. What failure mode is likely occurring? What should you fix?

3. Why is a "non-zero but declining" Avg Failures/Traj considered a positive behavioral signature?

4. What is the "glass ceiling" finding for CoT agents, and why can't DSPy optimization overcome it?

5. Why does decoupling measurement from deployment (via trace extraction) matter for production systems?