Step-by-Step Teaching

Step 1: Understanding Why Prototypes Break at Scale

The Prototype Mental Model

When you first build an AI application, the interaction looks like this:

User Request → Build Prompt → Call Model → Return Response

This works perfectly because each request is independent. If it fails, you just try again. Nothing carries over between calls.

What Changes at Production Scale

Now imagine running that same logic across 10,000 records simultaneously:

10,000 rows → 10,000 model calls → some succeed, some fail,
some hit rate limits, some timeout, worker crashes at row 8,432...

Suddenly you face questions your prototype never had to answer:

• What actually completed before the crash?
• Did row 8,432 fail, or did it succeed but fail to record success?
• Can you safely retry it without duplicating work?

Key Insight: You have stopped debugging AI behavior and started debugging distributed execution.

Quick Check ✓

Before moving on, ask yourself: Why does the single-request model break down at scale? (Answer: because it assumes independence, but large executions have shared state, partial progress, and interdependencies.)

---

Step 2: Identifying the Real Problem — Execution Has No Memory

The Row 8,432 Problem

Consider this scenario:

Background job processes rows 1 → 8,431 ✓
Worker crashes during row 8,432 ✗
System restarts...

Your instinct says: "Row 8,432 failed. Retry row 8,432."

But look deeper:

The failed row is not the real problem. It is simply where uncertainty becomes visible.

The Root Cause

The application has no explicit representation of the execution itself. It only has:

• Logs (incomplete)
• Database state (possibly inconsistent)
• Partial outputs (unreliable)

This is like trying to reconstruct a conversation from scattered sticky notes after someone threw them in the air.

---

Step 3: Defining the Unit of Work — Recovery Boundaries

The Two Wrong Extremes

Before you can fix failures, you must answer: What should the system actually recover?

❌ Too Broad:  Retry the entire 10,000-row job
               → Repeats thousands of successful calls
               → Wastes model quota
               → Risks overwriting correct outputs

❌ Too Granular: Retry every tiny instruction independently
                → Massive orchestration overhead
                → Hard to manage and reason about

The Right Approach: Hierarchical Execution Units

Model your execution as a hierarchy that mirrors how users think about the work:

Column Run (user's overall request)
    └── Batch / Row (meaningful execution unit)
            └── Activities (atomic operations)
                    ├── Resolve inputs
                    ├── Build prompt
                    ├── Call model
                    ├── Validate response
                    ├── Write output
                    └── Update progress

Why this works:

• A failure in Row 47 only restarts Row 47
• Rows 1–46 are untouched
• The user sees progress at a level they understand

Core Principle: The right retry boundary is the smallest meaningful unit of recoverable work.

Practical Exercise 🔧

Map this to something familiar: If you were processing 1,000 customer invoices, what would be your hierarchy? (Hint: Batch run → Individual invoice → Line item operations)

---

Step 4: Making Execution Durable — State That Survives Failures

The Fundamental Question

Once you have smaller recoverable units, a new problem emerges:

If a worker restarts, where does the next worker find out what already completed?

The Fragile Approach (What Most Systems Do)

Worker A processes rows 1-8,431
Worker A crashes
Worker B starts
Worker B reads logs + database + timestamps
Worker B tries to reconstruct what happened
Worker B makes assumptions...
Worker B gets it wrong in edge cases

This works when failures are rare. It becomes increasingly fragile as executions grow longer.

The Durable Approach (Reframing the Problem)

Instead of asking "How does a worker remember where it left off?", ask:

"What if the execution remembered its own history, independent of any worker?"

Traditional Model:          Durable Model:
┌─────────────┐            ┌─────────────────────────┐
│   Worker    │            │   Execution History      │
│  (holds     │            │   (persisted externally) │
│   state)    │            │                          │
│  crashes ✗  │            │   Worker A reads it ✓    │
└─────────────┘            │   Worker A crashes       │
                           │   Worker B reads it ✓    │
                           │   Execution continues ✓  │
                           └─────────────────────────┘

The worker becomes replaceable. The execution does not.

How This Works in Practice (Temporal Example)

Column Execution = Parent Workflow (durable, persisted)
    └── Row/Batch = Child Workflow (own retry behavior)
            └── Model Call = Activity (retry policy defined here)

If worker disappears:
→ Another worker picks up from persisted execution history
→ No reconstruction needed
→ No guessing about what completed

The Results This Produces

---

Step 5: Designing Smart Retry Strategies

Retries Must Match Execution Boundaries

Now that execution is durable and boundaries are defined, retries become precise:

Transient model timeout on Row 47:
→ Retry ONLY the activity that timed out
→ Row 47 resumes from last checkpoint
→ Rows 1-46 untouched ✓
→ Model quota not spent twice ✓

Validation failure on Row 23:
→ Isolated to Row 23
→ Other rows continue unaffected ✓

The Checkpoint Mechanism

Process Batch:
  ├── Row 1 → Success → Checkpoint ✓
  ├── Row 2 → Success → Checkpoint ✓
  ├── Row 3 → Failure → Retry (up to 10x, exponential backoff)
  │     ├── Attempt 2 → Success → Checkpoint ✓
  └── Row 4 → Success → Checkpoint ✓

If worker crashes after Row 2's checkpoint:
→ Resume from Row 3 (not Row 1)
→ Row 3 retries remaining attempts
→ Rows 1-2 never re-executed

Retry Policy Components

Every activity should define:

Rule Before Implementing Retries: Decide what your system is allowed to repeat before deciding how to repeat it.

---

Step 6: Observability — Making Progress Visible

The Problem with "Running..."

A single status indicator tells users and engineers nothing useful:

❌ "Running..." 
   → Is it progressing? Stalled? Retrying? 
   → Nobody knows.

Layered Progress Mirrors Layered Execution

Your observability model should match your execution hierarchy:

Worksheet Level:  "3 of 5 columns complete"
    └── Column Level:   "8,200 of 10,000 rows complete"
            └── Row Level:      "Row 47: Retrying (attempt 2/10)"
                    └── Cell Level:     "Value generated ✓ / Failed ✗ / Pending..."

Why this matters:

• Users can distinguish completed vs failed vs retrying
• Engineers can debug without reconstructing execution manually
• Progress is understandable, not mysterious

Principle: Progress visibility should mirror the same execution boundaries that define recovery. When they stay aligned, failures become understandable instead of mysterious.

---

Step 7: The Architectural Mindset Shift

Old Questions vs. New Questions

Most teams start AI architecture discussions here:

❌ Old Questions:
   "Which model should we use?"
   "How should we write the prompt?"
   "What temperature setting works best?"

Production AI demands these questions instead:

✓ New Questions:
   "What is the smallest meaningful unit of recoverable work?"
   "What state must survive a worker restart?"
   "Which operations are safe to retry?"
   "How will users know what completed vs. what needs attention?"

The Diagnostic Test

Imagine execution reaches row 8,432 before something unexpected happens.

---

Summary: The Complete Mental Model

PROTOTYPE AI                    PRODUCTION AI
─────────────────               ──────────────────────────────
Single request                  Long-running workflow
Independent calls               Coordinated execution
Stateless                       Durable state
Retry = start over              Retry = resume from boundary
"Did model respond?"            "What state survived?"
One status                      Layered observability
Worker = execution              Execution > any single worker

The Four Pillars of Reliable Production AI

1. EXECUTION HIERARCHY
   Define meaningful units at every level

2. DURABLE STATE  
   Execution history lives outside any single process

3. ALIGNED RETRIES
   Retry boundaries match execution boundaries

4. LAYERED OBSERVABILITY
   Progress visibility matches execution structure

---

Final Takeaway

The model generates intelligence. Your execution model determines whether that intelligence can be trusted.

Choosing a better model improves the quality of your outputs. Designing durable workflows determines whether those outputs survive to reach your users reliably at scale.