Evaluating Shopping Assistants with the PACE Framework

🎯 Learning Outcomes

After studying this material, you should be able to:

1. Explain why traditional product ranking evaluation is insufficient for modern shopping assistants
2. Define the PACE framework and its four components
3. Describe the PACES HOP benchmark dataset and its structure
4. Understand what PACE JUDGE is and how it works
5. Analyze why generic LLM judges fail at structured shopping assistant evaluation
6. Apply the concept of "task-matched output contracts" to evaluation design

---

📚 Step-by-Step Teaching

---

STEP 1: Understanding the Problem — Why Old Evaluation Methods Fall Short

The Old Way: Ranked Product Lists

Traditional shopping assistants worked like this:

User Query: "I need running shoes"
System Output: 
  1. Nike Air Zoom
  2. Adidas Ultraboost
  3. New Balance 880

Evaluation was simple: Did the right products appear? Were they ranked correctly?

The New Way: Structured Decision Support

Modern shopping assistants must do much more:

User Context: "I'm a 45-year-old with knee problems, 
               training for a 5K, budget $150"

System Must:
  ✓ Understand the shopper's personal context
  ✓ Synthesize product evidence (reviews, specs)
  ✓ Provide next-step guidance
  ✓ Deliver a coherent, consistent recommendation

Why This Creates an Evaluation Crisis

Key Insight: A response can be fluent and readable yet still fail in critical ways

The core problem: Existing benchmarks cover pieces of this problem but not the whole picture together

---

STEP 2: Introducing the PACE Framework

PACE defines the joint evaluation target — what a shopping assistant response must achieve simultaneously

P — Personalized
A — Actionable  
C — Compositional
E — Evidence-grounded

Let's break each one down:

---

🔵 P — Personalized

What it means: The response must reflect the specific shopper's context, needs, and constraints

Good Example:

"Given your knee sensitivity, I recommend shoes with extra heel cushioning and a wider toe box"

Bad Example:

"These are popular shoes with great reviews" (ignores the person entirely)

Why it matters: Generic advice is useless if it doesn't match who is asking

---

🟢 A — Actionable

What it means: The response must give the shopper clear next steps — not just information, but guidance on what to do

Good Example:

"Visit a specialty running store to get a gait analysis before purchasing, and ask specifically about stability features"

Bad Example:

"Running shoes come in many varieties" (informative but not actionable)

Why it matters: Decision support means helping someone decide and act, not just informing them

---

🟡 C — Compositional

What it means: The response has multiple components (e.g., product summary, pros/cons, recommendation rationale) that must be internally consistent

Think of it like a multi-part document:

Component 1: Product Overview
Component 2: Pros & Cons Analysis  
Component 3: Final Recommendation
Component 4: Next Steps

⚠️ All four must AGREE with each other

Bad Example (Compositional Failure):

• Component 2: "Excellent arch support"
• Component 3: "May not suit flat-footed runners" ← Contradiction!

Why it matters: Users read the whole response; contradictions destroy trust and usefulness

---

🔴 E — Evidence-grounded

What it means: Claims must be traceable to actual product evidence (reviews, specifications, test data)

Claim: "This shoe has superior cushioning"
Must be supported by: Specific review data, lab specs, 
                       or verified product information

Bad Example:

"This is widely considered the best cushioned shoe" (where does this come from?)

Why it matters: Shopping decisions involve real money; ungrounded claims mislead buyers

---

STEP 3: The PACES HOP Benchmark Dataset

Now that we know what to evaluate (PACE), we need a way to measure it

What is PACES HOP?

PACES HOP is a benchmark dataset that makes PACE measurable

Dataset Structure at a Glance

Total Records: 22,625 controlled entries

Each record contains:
├── Structured Persona (who the shopper is)
├── Auditable Evidence Pool (verifiable product information)
├── GOOD/BAD Label (overall quality judgment)
├── Gold Defect Family (what TYPE of problem exists)
└── Gold Defect Location (WHERE in the response the problem is)

Breaking Down Each Component

1. Structured Personas

Example Persona:
{
  "age": 45,
  "health_constraints": ["knee problems"],
  "goal": "5K training",
  "budget": "$150",
  "experience_level": "beginner"
}

This allows evaluation of whether the response truly used this information

---

2. Auditable Evidence Pools

Evidence Pool for Product X:
{
  "cushioning_rating": 8.5/10,
  "arch_support": "moderate",
  "user_reviews": ["great for long runs", "narrow fit"],
  "lab_test_results": {...}
}

"Auditable" means anyone can check whether a claim is supported

---

3. GOOD/BAD Labels Simple binary judgment — but the power comes from why something is BAD

---

4. Gold Defect Family + Location

This is what makes PACES HOP diagnostically powerful:

Analogy: It's like the difference between a doctor saying "you're sick" vs. "you have a bacterial infection in your left lung, specifically in the lower lobe"

The second is actionable — you know exactly what to fix and where

---

STEP 4: PACE JUDGE — Making Results Reportable

Having a benchmark is not enough — you need a judging protocol to apply it

What is PACE JUDGE?

PACE JUDGE is a training-free judging protocol with a structured output contract

Key Feature: "Training-free" = No model retraining required
             Works by changing HOW you prompt and structure outputs

The Structured Output Contract

Think of this as a required form that any judge must fill out:

PACE JUDGE Output Contract:
{
  "overall_quality": "GOOD" | "BAD",
  "persona_alignment_score": [0-1],
  "evidence_grounding_check": {
    "claim": "...",
    "source": "...",
    "supported": true | false
  },
  "cross_component_consistency": "PASS" | "FAIL",
  "defect_family": "personalization" | "grounding" | 
                   "actionability" | "composition" | null,
  "defect_location": "component_name + position"
}

Why "contract" language? Because the judge is obligated to fill every field — it cannot give a vague answer

---

Why Generic Judges Fail (And PACE JUDGE Doesn't)

This is a crucial finding from the research:

Generic LLM Judge Behavior:
Query: "Is this a good shopping recommendation?"
Output: "This is a well-written response that addresses 
         the user's needs with relevant product information. 
         Score: 7/10"

Problems:
❌ Cannot tell you WHERE the failure is
❌ Cannot tell you WHAT TYPE of failure occurred  
❌ Broad quality recognition ≠ diagnostic precision

PACE JUDGE Behavior:
Same Query → Structured Contract Forces:
✓ Persona-source check (did it use the persona?)
✓ Cross-component check (are sections consistent?)
✓ Grounding check (are claims supported?)
✓ Family/location closure (what failed and where?)

The Key Insight: Output Contract > Model Size

Critical Finding: You don't need a bigger or smarter model to evaluate better. You need a task-matched output contract that forces the right diagnostic questions.

Improvement came from:
✓ Structured output contract (PACE JUDGE)
✗ NOT from retraining the model
✗ NOT from using a more powerful backbone
✗ NOT from a simple scalar prompt ("rate 1-10")

---

STEP 5: Putting It All Together — The Full System

┌─────────────────────────────────────────────────────┐
│              THE PACE EVALUATION SYSTEM              │
├─────────────────────────────────────────────────────┤
│                                                      │
│  WHAT to evaluate ──→ PACE Framework                 │
│  (P + A + C + E)                                     │
│                                                      │
│  HOW to measure it ──→ PACES HOP Benchmark           │
│  (22,625 records with personas, evidence,            │
│   labels, defect family + location)                  │
│                                                      │
│  HOW to report it ──→ PACE JUDGE Protocol            │
│  (Structured output contract, training-free)         │
│                                                      │
└─────────────────────────────────────────────────────┘

---

STEP 6: Key Takeaways & Concept Connections

Summary Table

The Central Argument in Plain Language

Before PACE: "This response is pretty good" (useless for improvement)

After PACE: "This response fails on Personalization (P), specifically in the product recommendation section, because it ignores the shopper's knee constraint despite it being in the persona" (actionable, fixable)

---

🧪 Self-Check Questions

Test your understanding:

1. What are the four components of PACE? (Can you define each without looking?)

2. Why can a fluent response still fail PACE evaluation? (Think of three specific ways)

3. What makes PACES HOP "auditable"? (What specific feature enables this?)

4. Why is "defect location" important? (What analogy helps explain this?)

5. What is the key finding about PACE JUDGE vs. generic judges? (What does NOT need to change to improve evaluation?)

---

💡 Real-World Application

Scenario: You're building a shopping assistant for a furniture retailer. A customer says: "I need a sofa for a small apartment, I have two dogs, budget $800, and I prefer modern style."

Apply PACE to evaluate a response:

This is exactly what PACES HOP and PACE JUDGE are designed to evaluate systematically at scale.