Step-by-Step Study Material

Step 1: The Core Problem — The Trust Gap

What is easy vs. what is hard

EASY:  Getting an LLM to produce something useful
HARD:  Making that output trustworthy enough to ship

The Autonomy Spectrum

Think of AI systems on a leash of increasing length:

Key Insight: Each step further out on the leash required its own unique safeguards. Autonomy and safety engineering scale together.

---

Step 2: Project One — Data Quality Check Suggester

The Problem It Solves

Silent data corruption path:
Bad data lands → Models train on it → Dashboards report it
→ Problem surfaces DAYS LATER when someone notices a wrong number

Data quality checks are assertions that run after each ingestion:

• "This ID is never null"
• "This value stays in range"

Problem: Engineers write them by hand, so most datasets have few or none.

---

Why Use an LLM Instead of a Rules Engine?

A rules engine handles easy, mechanical cases:

• Column never null → add not-null check
• Jointly unique pair → add unique-key check

An LLM handles cases that depend on what columns mean:

Example: Two tenant ID columns on a payment table
- One written by upstream system
- One written by DoorDash's system
- Both always populated, always agree

LLM proposes: Flag any row where the two DISAGREE
LLM reasoning: IDs come from different systems and should reconcile
               A mismatch = routing error = could tag production events as test data

A rules engine would find that match plus dozens of accidental matches and could not distinguish a meaningful rule from a coincidence.

The Long Tail Problem:

Rules engine approach:
- Maintain lists of placeholder values worth flagging
- Maintain name patterns marking cross-system pairs
- Maintain dates that count as implausible
- Someone extends these lists EVERY TIME a new pattern appears

LLM approach:
- Avoids encoding most of that long tail up front

---

Critical Lesson: Data Representation Matters More Than Prompting

The worst bug in the project:

Profiler couldn't compute column statistics
↓
Left the field BLANK
↓
Blank stored same as measured ZERO
↓
Model read "zero non-null values" as "100% null"
↓
Proposed completeness check that would FAIL ON EVERY ROW

The model reasoned correctly over wrong input.

The fix:

BEFORE: Empty field = "0 non-null values"  ← ambiguous
AFTER:  Empty field = "not measured"       ← explicit

Core Principle: A lot of the trust work in an LLM system is in how you represent the data you feed the model, not in the model itself.

---

The Four Safeguard Layers

Layer 1: TIERED PROFILING
├── Cheap row count first
├── Sampling strategy matched to table size
└── Full scan (small) → bounded sample (large)

Layer 2: DETERMINISTIC POST-PROCESSING
├── Runs AFTER the model
├── Drops suggestions breaking known rules:
│   ├── Completeness check on nullable column → DROPPED
│   └── Check that fires on every row → DROPPED
└── Enforced in CODE, not in the prompt
    (A model won't apply the same rule the same way on every run)

Layer 3: HUMAN REVIEW
├── Every suggestion shown to engineer
├── Statistics behind each suggestion visible
├── Model reasoning available on request
├── Refine command accepts context ("this column is nullable by design")
└── Only accepted checks reach the draft PR

Layer 4: LLM-AS-A-JUDGE EVALUATION
Scores each suggestion on four axes:
├── Relevance to the column
├── Whether reasoning holds up
├── Whether assertion is specific enough to implement
└── Likelihood of false positives

---

Key Design Decision: Schema Over Prompting

The judge found a recurring problem: the tool picked the right column and explained why it mattered, then left the threshold undefined.

Wrong approach: Hope the model fills in the threshold
Right approach: Make threshold a REQUIRED OUTPUT FIELD in the schema

Principle: If you depend on a rule, enforce it in the output schema, not in the prompt.

---

Step 3: Project Two — Smart ML Dashboard

The Problem It Solves

ML observability data scattered across:
- Dozen internal dashboards
- Third-party tools
- Multiple data sources

Result:
- Hard for engineers to correlate
- Effectively unusable for agents
- No reasoning layer explaining what's happening and why

---

Key Design Decision 1: Dashboards as Documents

The challenge: How do you teach an LLM to write a dashboard? It can't see.

The insight: LLMs are very good at text.

Traditional approach:
Dashboard = Visual elements (LLM cannot verify layout or intent)

New approach:
Dashboard = YAML document (panels, queries, transformations, layout)

The three-phase parser:

Phase 1: SYNTAX CHECK
└── Reports line and column errors

Phase 2: SHAPE VALIDATION  
└── Per-issue errors pointing at exact paths in document

Phase 3: STRUCTURAL WALK
└── Catches what schema can't:
    ├── Duplicate panel IDs
    └── References to undeclared sources

Every failure → uniform error shape
Agent gets same structured feedback as human editor
Agent can iterate on its own output

---

Key Design Decision 2: One Data Shape for Every Source

Inspired by observability platform data models, all data presented as DataFrames:

DataFrame = list of named, typed fields
          + row-count metadata  
          + per-field labels and config

Source plugin = async function that returns DataFrames

Three properties this buys:

Property 1: PANEL INDEPENDENCE
└── Timeseries panel renders ANY DataFrame with time field + number fields
    Same panel, any backend

Property 2: UNIVERSAL TRANSFORMATION COMPOSITION
└── Each transformation = pure function from frames to frames
    Runs in order after fetch
    None know which source produced the frames

Property 3: SINGLE LEARNING TARGET
└── The LLM only has to learn ONE data model

---

Key Design Decision 3: Context-Efficient Prompting

The challenge: Encoding all panel configurations exhausts the context window.

Panel configurations include:
- Field options
- Units  
- Thresholds
- Mappings
- And more...

Solution: "Ask only for what you need"

Initial agent context:
└── Concise descriptions of panels and data sources only

When more detail needed:
└── Agent requests it through tool calls

Result:
├── Reduced context usage
└── Improved dashboard generation and debugging

Enabled by registry design:

Registry contains:
├── Machine-readable schema (for validation and codegen)
└── Human-readable documentation (for the agent)

Adding new panel = single file that registers itself
Agent gains ability to use it immediately
No prompt changes, no retraining

---

Key Design Decision 4: Dashboard as Signal

Traditional dashboard: Shows data

Smart ML Dashboard:
├── Shows data
├── Enables AI debugging within panels
├── Declares what the user CARES ABOUT
└── Enables proactive agentic reports before user asks

Overview tab:
└── Agent's analysis without requiring user to read the data at all

---

Step 4: Project Three — Automated Root Cause Analysis

The Problem It Solves

Current state:
Detection of failures → SOLVED (per-axis judges, trend monitoring)
Diagnosis of failures → NOT SOLVED

Diagnosis still means:
Engineer reads transcript → Forms hypothesis → Confirms against logs and commits

Scale problem:
Thousands of transcripts/week
2% flag rate = substantial manual review burden
Review cost scales with usage
Team size does NOT scale with usage

---

Why Diagnosis Is Hard

Three properties making each investigation expensive:

Property 1: FAULT ORIGIN DECOUPLED FROM SYMPTOMS
└── Failure in final turn can originate several agents upstream
    Reading the trace alone does not localize it

Property 2: DENSE PRODUCTION GUARDRAILS
└── Policy enforced across:
    ├── System prompts
    ├── Policy engines
    └── Tool-layer checks
    Two rules can silently conflict
    Hard to tell if rule was missing, wrong, overridden, or not followed

Property 3: HETEROGENEOUS FAILURE MODES
└── Execution faults: truncation, premature termination, orphaned tool calls
    Semantic faults: contradictory context, malformed handoff, sampling variance
    These share almost nothing diagnostically
    Stochastic execution means failures often can't be reproduced

---

Prerequisites for a Self-Healing Loop

OBSERVABILITY → What to investigate
└── Full-fidelity capture of input, tool calls, steps, outcome

GRADING → Why to investigate
└── Per-axis metrics and judges tied to product specification
    Multi-agent failures don't lie on single axis
    One aggregate score rarely captures the failure

MONITORING → When to investigate
└── Trend monitoring over metrics
    Detection triggered by anomaly, not on demand

---

The Pipeline Architecture

MINING STAGE                    TUNING STAGE
(covered in detail)             (proposed fix + regression check)
        ↓
Three sequential phases:

PLANNER
├── Reads: transcript, judge verdicts, execution signals
├── Does NOT conclude
└── Emits: hypotheses + questions to confirm/refute them
        ↓
WORKERS (fan-out)
├── Each inspects: source code, git logs, OR one implicated prompt
├── Returns: verbatim evidence for assigned questions ONLY
└── May NOT conclude (prevents self-confirming pipeline)
        ↓
MANAGER
├── Adjudicates evidence
└── Produces per-session diagnosis:
    ├── Originating fault
    ├── Propagation path
    ├── Responsible agent
    ├── Plausible fix location
    └── Implicated prompt span quoted verbatim

Why split the work? Supplying all evidence in one context window yields a confident but unfalsifiable summary. Splitting forces each agent to stay in its lane.

---

The Output Schema Design

Fixed schema contains:
├── Causal chains
├── Verbatim evidence
└── Categorization fields from published taxonomy

Schema serves TWO purposes:
1. Constrains what the Manager MUST justify
2. Makes findings comparable and auditable across sessions

Comparable findings → can be embedded and clustered
Clustered findings → proposed fix targets a FAMILY of sessions, not one trace

---

Evaluation: Bespoke Pipeline vs. Off-the-Shelf Agent

Four conditions tested:

Condition 1: Free agent, no schema
Condition 2: Free agent + schema constraint
Condition 3: Multi-agent, no schema  
Condition 4: Custom pipeline (full structure)

Key findings:

Window 1 Results:
├── Custom pipeline: led on output quality and groundedness
├── Simple free agent: led on equal-weighted 10-axis score
└── Two intermediate conditions: fixated on headline anomaly
    (provider-side latency noise no code change could address)

Lesson: Splitting work across agents ≠ deeper investigations
        Giving each agent a TIGHTLY SCOPED TASK = deeper investigations

Window 2 Results (after improvements):
├── Custom RCA + semantic clustering: best overall
│   ├── Output quality ✓
│   ├── Groundedness ✓
│   ├── Sampling robustness ✓
│   └── Reproducibility ✓
└── Free agent: kept novelty, portability, lower cost per run

---

Three Transferable Conclusions

Conclusion 1:
Off-the-shelf agent harnesses CAN already root-cause agentic failures today

Conclusion 2:
Customized structure adds provenance, specificity, thematic stability
at the cost of portability

Conclusion 3:
Schema helps at REPORTING step
Schema HURTS at EXPLORATION step
(intermediate conditions showed this clearly)

---

Step 5: The Unifying Thread

All three projects independently arrived at the same four safeguards:

Safeguard 1: DETERMINISTIC GUARDRAILS OUTSIDE THE MODEL
└── Rules enforced in code, not in prompts
    A model won't apply the same rule the same way on every run

Safeguard 2: DATA REPRESENTATION THE MODEL CAN'T MISREAD
└── Explicit nulls instead of magic numbers
    "not measured" instead of "0 non-null values"
    One data shape for every source

Safeguard 3: OUTPUT SCHEMA CONSTRAINING WHAT IT MUST JUSTIFY
└── Required fields instead of hoped-for fields
    Fixed handoff contracts between agents
    Structured findings with verbatim evidence

Safeguard 4: EVALUATION HARNESS MAKING "DID THIS HELP" A NUMBER
└── LLM-as-a-judge scoring on defined axes
    Before/after comparison on prompt or model changes
    Multi-window, multi-replicate evaluation

---

Summary Table

---

Key Takeaways to Remember

1. Generating output is easy. Making it trustworthy is the engineering work.

1. The representation of data fed to the model matters as much as the model itself.

1. Any rule you depend on must be enforced in code, outside the model.

1. Schema constrains what the model must justify — use it at reporting, not exploration.

1. "Did this help" must be a number, not a feeling.

1. Autonomy and safety engineering must scale together.