After studying this material, students should be able to:

1. Explain why automated scenario synthesis for AI agent evaluation is necessary
2. Describe how tool specifications encode semantic information usable for test generation
3. Understand the Agent Seer pipeline architecture and its key components
4. Identify the primary factors affecting evaluation quality
5. Recognize the limitations of coarse-grained evaluation metrics

---

Step-by-Step Teaching

Step 1: The Core Problem — Why Is This Hard?

The Challenge of Evaluating AI Tool-Using Agents

Imagine you built an AI assistant that can call external tools like:

• A weather API
• A calendar system
• A database query tool

How do you test if it works correctly?

Traditionally, you would:

Manual Approach:
Human expert → writes test scenarios → runs tests → evaluates results

Why This Fails at Scale

Key Insight: We need a way to automatically generate realistic test scenarios

---

Step 2: The Key Observation — Hidden Information in Specifications

What Is a Tool Specification?

A tool specification is a structured description of what a tool does. Think of it like a job description for a function.

Example Tool Specification:

{
  "name": "get_weather",
  "description": "Retrieves current weather for a given location",
  "parameters": {
    "city": {
      "type": "string",
      "description": "Name of the city"
    },
    "units": {
      "type": "string",
      "enum": ["celsius", "fahrenheit"],
      "description": "Temperature unit"
    }
  }
}

What Information Is Already Encoded Here?

Tool Specification Contains:
├── Function Name → WHAT the tool does
├── Natural Language Description → HOW it should be used
└── Parameter Schema → WHAT inputs it expects

Agent Seer's Core Insight:

This information is sufficient to synthesize realistic test scenarios — without running the tool, without examples, and without domain experts

---

Step 3: What Is MCP (Model Context Protocol)?

Understanding the Foundation

Model Context Protocol (MCP) is a standardized format for describing tools that AI agents can use.

Think of it like:

MCP : AI Tools  =  USB Standard : Electronic Devices

Just as USB standardizes how devices connect, MCP standardizes how tools are described to AI agents.

Why MCP Matters for Agent Seer

• It provides a consistent input format
• Agent Seer only needs one MCP specification to start
• No live tool access required
• No domain-specific tuning needed

---

Step 4: The Agent Seer Pipeline — How It Works

Overview

INPUT: Single MCP Specification
         ↓
    [Step 1] Schema Enrichment
         ↓
    [Step 2] Scenario Generation
         ↓
    [Step 3] Multi-Turn Dialogue Expansion
         ↓
OUTPUT: Realistic Evaluation Scenarios

Step 4.1 — Schema Enrichment

What happens: Raw schemas are enhanced with additional semantic context

Why it's needed:

Raw Schema:          →    Enriched Schema:
"city: string"            "city: string (e.g., 'London', 
                           'New York') — required for 
                           location-based queries"

The enrichment fills in implicit knowledge that humans understand but machines need explicitly stated.

Step 4.2 — Graded Scenario Generation with Synthetic Tool Outputs

What happens: The system creates test scenarios at different difficulty levels

Graded means:

Synthetic Tool Outputs are fake but realistic responses:

Real tool output:    {"temp": 22, "condition": "sunny"}
Synthetic output:    {"temp": 19, "condition": "cloudy"}  ← Generated, not real

This allows testing without actually calling live APIs.

Step 4.3 — Multi-Turn Dialogue Expansion

What happens: Single scenarios expand into realistic conversations

Why multi-turn matters:

Turn 1: User: "What's the weather in Rome?"
Turn 2: Agent: [calls weather tool] "It's 24°C and sunny"
Turn 3: User: "Should I bring an umbrella tomorrow?"
Turn 4: Agent: [calls forecast tool] "No rain expected"
Turn 5: User: "Great, book me a restaurant with outdoor seating"
Turn 6: Agent: [calls restaurant tool] ...

This tests whether the agent:

• Maintains context across turns
• Calls tools in the right sequence
• Uses previous results correctly

---

Step 5: How Quality Is Measured

Two Key Metrics

Evaluation Quality
├── Tool-Calling Correctness
│   ├── Correct tool selected? (name match)
│   └── Correct arguments passed? (argument accuracy)
└── Conversational Coherence
    └── Does the dialogue flow naturally?

The Hidden Failure Mode

Coarse-grained metric (name match):

Expected: get_weather(city="Rome", units="celsius")
Actual:   get_weather(city="Roma", units="fahrenheit")

Name Match Score: ✅ PASS (correct tool name)
Reality:          ❌ FAIL (wrong arguments)

Critical Finding: Argument value accuracy is the dominant failure mode — but it's invisible to simple name-matching metrics

This is like grading a math test only on whether students wrote the right formula, ignoring whether they calculated the right answer.

---

Step 6: Key Findings from Evaluation

Testing Across 7 MCP Specifications

Agent Seer was tested on 7 different tool specifications across diverse domains with varying numbers of tools.

Finding 1: Parameter Complexity Matters Most

Quality Variation Explained By:

Parameter Schema Complexity  ████████████████  (STRONGEST factor)
Tool-Suite Size              ████              (smaller, separate factor)

What this means:

• A specification with simple parameters (strings, integers) → Higher quality output
• A specification with complex parameters (nested objects, conditional requirements) → More errors
• Having many tools matters less than how complex each tool's parameters are

Analogy: Teaching someone to use 20 simple tools is easier than teaching them to use 5 tools with complicated settings.

Finding 2: Complete Coverage on Small/Medium Specs

Small Specs:   ✅ All tools covered
Medium Specs:  ✅ All tools covered  
Large Specs:   ⚠️  Some gaps appear

---

Step 7: Connecting to the Broader Research Landscape

Related Work Mentioned

PORTool addresses a related problem:

Problem: When an agent uses multiple tools to solve a task,
         which step caused success or failure?

Solution: Importance-aware rewards that credit individual
          tool-use decisions, not just final outcomes

Reinforced Agent addresses another gap:

Problem: Current evaluation happens AFTER execution
         (post-hoc) — errors found too late

Solution: Real-time feedback DURING tool-calling execution

How They Connect

Agent Seer    → Generates test scenarios automatically
PORTool       → Trains agents better using those scenarios  
Reinforced    → Evaluates agents in real-time during execution
Agent

---

Summary: Complete Mental Model

THE BIG PICTURE:

PROBLEM: Testing AI tool-using agents is expensive and doesn't scale

INSIGHT: Tool specifications already contain enough information
         to generate tests automatically

SOLUTION (Agent Seer):
  Input:  MCP Specification (just the description)
  
  Pipeline:
  1. Enrich schemas with semantic context
  2. Generate graded scenarios + fake-but-realistic outputs
  3. Expand into multi-turn conversations
  
  Output: Realistic evaluation scenarios

KEY FINDINGS:
  ✦ Parameter complexity = biggest quality predictor
  ✦ Tool count = secondary factor
  ✦ Argument accuracy = hardest to get right
  ✦ Name-match metrics alone are insufficient

---

Quick Self-Check Questions

1. Why can't we simply write test scenarios by hand for all AI tools?
2. What three types of information does a tool specification contain?
3. What does "graded scenarios" mean in the Agent Seer pipeline?
4. Why is argument value accuracy considered a "hidden" failure mode?
5. Which factor most strongly predicts quality variation — tool count or parameter complexity?