Step-by-Step Teaching
Step 1: What Is Grounded Reasoning?
Start with the core concept.
Grounded reasoning means answering complex questions using evidence from real documents โ not from memorized training data.
Think of it like an open-book exam versus a closed-book exam:
| Closed-Book (Standard LLM) | Open-Book (Grounded Reasoning) |
|---|---|
| Relies on memorized knowledge | Must find evidence in documents |
| Fast but can hallucinate | Slower but verifiable |
| Works on general questions | Required for enterprise/proprietary data |
Why does this matter? Enterprises have massive private document collections โ financial reports, legal contracts, internal policies. AI must reason over those specific documents, not guess from training data.
Key Insight: 18.8% of questions went unsolved by every team, showing this remains an unsolved, economically important problem.
Step 2: Understanding the Competition Structure
Before analyzing strategies, understand the testing environment.
The Grounded Reasoning Cup was designed to answer one critical question:
"Do performance improvements on a benchmark generalize to new, real-world tasks?"
How It Worked:
Phase 1 (2 months): Teams trained/optimized on OfficeQA benchmark
โ
Phase 2 (Competition Day): Teams applied agents to NEW benchmark (OfficeQA Pro V2)
โ
Result: Tests TRUE generalization, not just memorization
Why This Design Matters:
- Teams could not overfit to the competition data
- Forces agents to be genuinely capable, not just benchmark-tuned
- Mirrors real-world deployment conditions
Step 3: The Four Core Building Blocks of Winning Agents
All top teams shared these components, regardless of their specific strategy:
๐น Building Block 1: Document Preprocessing
Problem: Raw PDFs and documents are messy. Models struggle with unstructured text.
Solution: Convert documents into clean, structured formats before the agent works on them.
Raw PDF โ Parser (e.g., ai_parse) โ Structured Markdown/Text โ Agent
Why it matters: Stanford's agent knew when to use parsed text vs. fall back to the original PDF. UMass built a metadata catalog for rapid filtering. This preprocessing step alone contributed significantly to performance gains.
๐น Building Block 2: Targeted Retrieval
Problem: Enterprise document collections are large. Feeding everything to a model is expensive and noisy.
Solution: Build smart search systems that find only the relevant evidence for each question.
Think of it like a librarian vs. reading every book:
- Bad approach: Give the model all documents
- Good approach: Retrieve only the 2-3 most relevant sections
UMass's metadata catalog enabled this rapid, targeted search โ contributing to their 4-minute average answer time vs. the team average of 8.5 minutes.
๐น Building Block 3: Structured Tool Use
Problem: LLMs are unreliable at precise calculations, unit conversions, and data lookups.
Solution: Give agents specialized tools for specific subtasks rather than asking the model to do everything.
Stanford built 100+ reusable skills, including:
- Table localization tools
- Financial terminology clarification
- Unit scaling handlers
- Answer formatting procedures
Mental Model: Think of this like giving a worker a toolbox. A hammer for nails, a wrench for bolts โ the right tool for each job.
๐น Building Block 4: Answer Verification
Problem: Agents make mistakes. A single wrong step can cascade into a wrong answer.
Solution: Add a verification layer that checks the answer before submission.
Stanford used a second Claude agent as a verifier that:
- Re-extracted intermediate values
- Checked data lineage (were revised numbers used correctly?)
- Handled unit scaling errors
- Patched calculations when discrepancies were found
Step 4: Comparing the Three Winning Strategies
Now let's see how each team combined these building blocks differently:
๐ฅ Stanford โ Accuracy-First with Adaptive Verification
Core Philosophy: Build a deep playbook; prioritize correctness
Question โ 100+ Skills Playbook โ Claude Opus Agent โ Verifier Agent โ Answer
Key Innovation: Traced every wrong answer during training back to its exact failure point, then turned that failure into a reusable skill.
Trade-off Demonstrated:
- Verifier ON โ Higher accuracy, slower speed
- Verifier OFF โ Faster, more speed bonuses
- Stanford switched the verifier on and off during competition based on round needs
Result: Won by accuracy on hardest questions in the final round
๐ฅ UMass โ Speed-First with Parallel Scaling
Core Philosophy: Fast is valuable; use parallelism to compensate for individual agent errors
Question โ Metadata Catalog Search โ 3 Parallel Agents โ Verifier Selects Best โ Answer
Key Innovation: Test-time scaling through parallelism โ run multiple agents simultaneously, pick the best answer
Trade-off Demonstrated:
- Fastest average correct answer time: 4 minutes
- Earned 36 speed bonuses (worth 0.25 pts each) vs. Stanford's 16
- Led by 10.25 points at halftime
- Lost on hardest final-round questions where accuracy mattered more
๐ฅ Yale โ Robustness-First with Diverse Architectures
Core Philosophy: No single agent should be a single point of failure
Question โ 4 Independent Arms (2 ReAct + 2 Planner-Verifier) โ Meta-Verifier โ Answer
โ
(Fallback: Majority Vote)
Key Innovation: Decorrelated failure modes โ different agent architectures fail in different ways, so combining them reduces overall error rate
Two Agent Types Used:
| ReAct Agents | Planner-Verifier Pipeline |
|---|---|
| Autonomous reasoning | Structured, step-by-step |
| Flexible | More controlled |
| Gemini Pro + Flash | Gemini Pro throughout |
Step 5: The Central Lesson โ Systems Thinking Over Model Choice
This is the most important conceptual takeaway:
Winning is not about picking the best model. It's about building the best system around the model.
The Systems Thinking Framework:
[Document Preprocessing] โ [Targeted Retrieval] โ [Model Reasoning] โ [Verification] โ [Answer]
โ โ โ โ
How clean is Can we find Can the model Did we catch
the input? the right evidence? reason correctly? the mistakes?
Each stage must work well. A brilliant model with poor retrieval will fail. A fast model with no verification will make costly errors.
Analogy: Think of it like a surgical team, not a solo surgeon:
- Preprocessing = Prep nurse (clean instruments)
- Retrieval = Anesthesiologist (right conditions)
- Model = Surgeon (core task)
- Verification = Attending physician (double-check)
Step 6: Broader Lessons to Remember
Synthesize everything into transferable principles:
| Lesson | What It Means |
|---|---|
| Generalization is hard | Optimizing for one benchmark doesn't guarantee real-world performance |
| Speed vs. accuracy is a real trade-off | UMass won on speed; Stanford won on accuracy โ both matter depending on context |
| Parallelism is a valid scaling strategy | Running multiple agents and selecting the best answer improves reliability |
| Failure analysis drives improvement | Stanford's 100+ skills came from systematically studying mistakes |
| Document parsing is foundational | ai_parse contributed 24 points of improvement over baseline โ infrastructure matters |
| Headroom remains | 18.8% of questions unsolved by everyone = significant room for future progress |
Quick Review: Check Your Understanding
Q1: Why was testing on a new benchmark on competition day important?
To test genuine generalization, not benchmark memorization
Q2: What is the difference between UMass's and Yale's parallelism strategies?
UMass ran 3 identical agents for speed; Yale ran 4 architecturally different agents for robustness
Q3: Why did Stanford turn their verifier off mid-competition?
It was causing latency that cost them speed bonuses; they adapted in real time
Q4: What does "decorrelated failure modes" mean in Yale's approach?
Different agent architectures fail in different ways, so combining them reduces the chance all fail simultaneously
Q5: Name the four core building blocks shared by all top teams.
Document preprocessing, targeted retrieval, structured tool use, answer verification