After studying this material, you should be able to:

1. Explain the "evaluation-first" approach to building AI agents
2. Describe the dual-loop model (development + production) and how they connect
3. Identify the key components of a production-grade AI agent evaluation framework
4. Apply evaluation strategies to real-world agentic systems
5. Analyze why traditional monitoring fails for AI agents

---

Step-by-Step Study Material

Step 1: Why Evaluation-First Matters (The Problem)

The Core Challenge

Imagine running a customer support system handling 100,000 AI tickets per day.

1% error rate = 1,000 bad customer experiences DAILY

Traditional software thinking says:

"If the server is up and returning responses, everything is fine."

This is completely wrong for AI agents.

An agent can have:

• ✅ Zero server errors
• ✅ 100% uptime
• ❌ Repeating the wrong answer to every frustrated customer

Why AI Agents Fail Differently

Unlike simple software, an AI agent is a multi-step workflow:

Customer Query
      ↓
[Classify Intent]
      ↓
[Retrieve Knowledge]
      ↓
[Analyze Inputs]
      ↓
[Reason Through Decision]
      ↓
[Call Tools/APIs]
      ↓
[Generate Response]

Failure can happen at ANY step, not just the final answer.

This is called the Assurance Gap — the space between "the system is running" and "the system is working correctly."

Real-World Pressures That Expose This Gap

Key Insight: "Just ship the agent" works until it breaks at scale. Evaluation must be infrastructure, not an afterthought.

---

Step 2: The Dual-Loop Model (The Solution Architecture)

This is the central concept of the entire framework.

┌─────────────────────────────────────────────────────┐
│              DEVELOPMENT LOOP                        │
│                                                      │
│  Code Change → Golden Dataset Test → Scorers →      │
│  Quality Gate → Auto-Deploy (if passed)              │
└──────────────────────┬──────────────────────────────┘
                       │ Quality Gate
                       ↓
              [PRODUCTION ENVIRONMENT]
                       │
                       ↓
┌─────────────────────────────────────────────────────┐
│              PRODUCTION LOOP                         │
│                                                      │
│  Live Traffic → Traces → Online Evaluation →         │
│  Alerts → Failure Captured → Fed Back to Dev Loop   │
└─────────────────────────────────────────────────────┘

How the Loops Work Together

Development Loop asks: "Is this agent ready for production?"

• Tests against known examples
• Blocks bad changes before customers see them
• Automated regression on every change

Production Loop asks: "Is the live agent still working correctly?"

• Monitors real traffic continuously
• Catches new failure modes
• Feeds failures back into development

Key Insight: Every production failure becomes a test case. The system gets smarter over time automatically.

---

Step 3: The Five Building Blocks of the Framework

Building Block 1: Tracing (Observability)

What it is: Recording every step of every agent interaction

What gets captured:

• Prompts sent to the model
• Model completions/responses
• Documents retrieved
• Tool calls made
• Latency at each step
• Decision paths taken

How it works technically:

# One line enables automatic tracing
mlflow.<library>.autolog()

# Custom spans for specific steps
@mlflow.trace
def my_custom_step():
    # your code here

Why it matters: Without traces, you only see input and output. With traces, you see everything in between — where failures actually occur.

---

Building Block 2: Evaluation Pillars and Gates

What it is: Translating stakeholder needs into measurable thresholds

Instead of vague debates about quality, each stakeholder defines what success means numerically:

┌─────────────────────────────────────────────────────┐
│                EVALUATION PILLARS                    │
├──────────────────┬──────────────────────────────────┤
│ Customer         │ Response accuracy > 94%           │
│ Experience       │ Empathy score > threshold         │
│                  │ Resolution rate > X%              │
├──────────────────┼──────────────────────────────────┤
│ Operational      │ Automation rate > 80%             │
│ Efficiency       │ Escalation rate < Y%              │
│                  │ Latency P95 < Z ms                │
├──────────────────┼──────────────────────────────────┤
│ Risk &           │ Fraud detection rate > threshold  │
│ Compliance       │ Zero prompt injection pass-through│
├──────────────────┼──────────────────────────────────┤
│ Financial        │ Cost per ticket < $X              │
│ Impact           │ Refund accuracy within policy     │
└──────────────────┴──────────────────────────────────┘

Key principle: All gates must pass before deployment. This turns subjective quality debates into objective, shared contracts.

---

Building Block 3: The Golden Dataset

What it is: A curated collection of test examples that represents the full range of real-world scenarios

Requirements for a good golden dataset:

• Covers all intent types and edge cases
• Includes adversarial examples (fraud attempts, prompt injection)
• Has verified correct answers
• Contributed to by ALL stakeholders (including security team)
• Continuously updated with production failures

How it evolves over time (Zepto's journey):

Month 1:  500 examples  → 8-point dev/prod accuracy gap
Month 3:  2,000 examples → 2-point gap
Month 6:  5,247 examples → 0.4-point gap

The 10x Rule:

Every hour spent improving the golden dataset saves ~10 hours of production debugging.

Why the gap matters: If your agent scores 94% on development tests but only 86% in production, your dataset is missing 8 points worth of real-world scenarios.

---

Building Block 4: Scorers (The AI Jury)

What it is: Automated systems that evaluate agent outputs along multiple dimensions

Three types of scorers:

┌─────────────────────────────────────────────────────┐
│                    AI JURY                           │
├──────────────────┬──────────────────────────────────┤
│ Rule-Based       │ Use when: deterministic logic     │
│ Scorers          │ Example: Did agent call the       │
│                  │ correct API? (yes/no)             │
├──────────────────┼──────────────────────────────────┤
│ ML Model         │ Use when: pattern recognition     │
│ Scorers          │ Example: Sentiment classification │
├──────────────────┼──────────────────────────────────┤
│ LLM-Based        │ Use when: human-like judgment     │
│ Scorers          │ Example: Was the response         │
│                  │ empathetic? Was it grounded?      │
└──────────────────┴──────────────────────────────────┘

Best practices:

• Use LLM scorers only where necessary (they're expensive)
• Calibrate LLM judges against human labels (target: 80-90% agreement)
• Use multiple judges for high-stakes decisions (consensus rules)
• Use simple rules where deterministic logic is sufficient

---

Building Block 5: Stratified Sampling

The problem: Evaluating 100% of traffic is too expensive. But random 10% sampling misses most edge cases.

The solution: Sample strategically based on risk:

HIGH sampling rate for:
├── High-value customers
├── New or recently changed features
├── Negative sentiment detected
├── High escalation risk signals
└── Image-based or fraud-prone interactions

LOWER sampling rate for:
└── Routine, low-risk interactions

Results of this approach:

---

Step 4: The Agent Architecture That Makes Evaluation Work

Why Architecture Matters for Evaluation

A well-designed agent architecture makes evaluation easier by being decomposable — you can measure each piece separately.

Vertical vs. Horizontal Agents

                    Customer Query
                          ↓
              [Orchestrator / Router]
                          ↓
         ┌────────────────┴────────────────┐
         ↓                                 ↓
   VERTICAL AGENTS                  HORIZONTAL AGENTS
   (Specialists)                    (Oversight Layers)
   
   • WIMO Agent                     • Fraud Detection Agent
     (Where Is My Order)            • Image Validation Agent
   • Refund Agent                   • Quality Assessment Agent
   • Cancellation Agent             • Compliance Agent
   • Expiry/Quality Agent

Vertical agents = Deep expertise in one domain

• Measured on domain-specific metrics
• Example: WIMO intent classification F1 score

Horizontal agents = Cross-cutting concerns applied to all interactions

• Measured on cross-cutting metrics
• Example: Fraud detection precision across all ticket types

Why this separation matters for evaluation:

• Each vertical agent can be tested in isolation
• Horizontal agents can be evaluated independently
• Problems are easier to locate and fix

---

Step 5: The Development Loop in Practice

How a Typical Change Flows

Engineer makes a change
         ↓
Automated regression triggered
         ↓
Change tested against Golden Dataset
         ↓
Scorers evaluate across all pillars
         ↓
    ┌────┴────┐
    ↓         ↓
All gates   Any gate
passed?     failed?
    ↓         ↓
Auto-deploy  Block + Alert
to production engineer

Automated Prompt Optimization

Instead of manually writing and testing prompts:

1. Register initial prompt in MLflow
         ↓
2. Generate multiple prompt variants automatically
         ↓
3. Score all variants against golden dataset
   (using same scorers that gate deployment)
         ↓
4. A/B evaluate automatically
         ↓
5. Deploy best-performing variant

Cost-aware optimization trick: Use a powerful (expensive) model to generate candidate prompts, but use a cheaper model to score them. This keeps the search process affordable.

---

Step 6: The Production Loop in Practice

Alert Tiers

CRITICAL (checked every 5 minutes):
├── Intent accuracy drops below threshold
├── Groundedness violations detected
├── High escalation risk spike
└── P95 latency breach

HIGH/MEDIUM (monitored continuously):
├── Empathy score degradation
├── Cost per ticket spike
├── Tool failure rate increase
├── CSAT trend decline
├── Fraud detection rate change
└── Multimodal processing latency

Real Example: The Stuck Rider Problem

What happened:

• Rider was stuck in traffic
• Agent kept responding: "Your order is arriving in 10 minutes"
• Customer asked again → same response
• Customer asked again → same response

How evaluation caught it:

• Token-counter scorer detected repetitive responses
• Warning scorer flagged high escalation risk
• Traces showed stationary rider with unchanged ETA
• Detection time: 5 minutes

What changed as a result:

Old behavior: Repeat cached ETA indefinitely

New rule: If rider stationary > 10 minutes:
├── Give honest status update
└── Proactively offer cancellation + full refund

New features unlocked by this insight:

• Cancel-on-delay feature
• Proactive cancellation pitch during rider shortages
• Auto-cancel if no rider assigned within time window
• No-questions-asked cancellation for high-value customers

---

Step 7: Advanced Concepts

Multimodal Evaluation (Images)

When customers submit photos of damaged/expired products, evaluation becomes complex:

Challenge 1: Human disagreement

• Same mushroom photo: one rater gives 2/5, another gives 3/5
• Solution: Measure disagreement with Cohen's Kappa — this sets your reliability ceiling
• No AI model can be more consistent than the humans it learns from

Challenge 2: AI plays it safe

• AI clustered scores at 3/5 to avoid hard calls
• Humans peaked at 4/5 and 5/5
• Solution: Match the shape of the human score distribution, not just minimize average error

Challenge 3: Edge cases

• Curdled milk (packaged good but needs fresh-produce judgment)
• Taste/smell complaints (photo can't show these)
• Solution: Route these to separate handling paths

Fraud Detection Pipeline

For refund abuse (fake photos, edited images, reused images):

Image submitted
      ↓
Preprocessing checks
(blur, brightness, resolution)
      ↓
OCR validation
      ↓
Jury of 3 vision models
(consensus rules)
      ↓
Additional checks:
├── Blur detection
├── Screenshot detection  
├── Duplicate image detection
├── Image vs. SKU matching
├── Image vs. stated reason check
└── Proof-of-delivery validation
      ↓
Auto-approve OR Route to human review

---

Step 8: Results and Key Principles

Zepto's Outcomes

Core Principles to Remember

1. Evaluation is infrastructure, not a final check

   • Build it first, not last

2. Traces are your foundation

   • You cannot evaluate what you cannot observe

3. Golden datasets compound in value

   • Every production failure makes future versions more robust

4. Use the right scorer for the right job

   • Rules for deterministic checks, LLMs only for judgment calls

5. Sample strategically, not uniformly

   • Risk-based sampling finds more problems at lower cost

6. Architecture and evaluation co-design

   • Decomposable agents are measurable agents

7. The dual loop creates a self-improving system

   • Production failures automatically strengthen development tests

---

Quick Reference Summary

EVALUATION-FIRST AI AGENTS
│
├── WHY: Assurance gap between "running" and "working correctly"
│
├── WHAT: Dual-loop model
│   ├── Development Loop: Test before production
│   └── Production Loop: Monitor in production
│
├── HOW: Five building blocks
│   ├── 1. Tracing (observe everything)
│   ├── 2. Evaluation Pillars (measure what matters)
│   ├── 3. Golden Dataset (source of truth)
│   ├── 4. Scorers/AI Jury (automated evaluation)
│   └── 5. Stratified Sampling (efficient coverage)
│
├── ARCHITECTURE: Vertical + Horizontal agents
│   ├── Vertical: Domain specialists (measurable in isolation)
│   └── Horizontal: Cross-cutting oversight layers
│
└── RESULTS: 80%+ automation, 65% cost reduction, <1 month payback