After studying this material, students should be able to:

1. Explain why traditional human-scaled maintenance fails for large open-source projects
2. Describe the architecture and design principles of an AI-powered software factory
3. Identify how to align automation depth with risk level
4. Trace the complete lifecycle of a change through a software factory pipeline
5. Apply key design principles when planning their own automated development systems

---

Step-by-Step Teaching

Step 1: Understanding the Problem — Why Human Scaling Breaks

The Core Challenge

Imagine you run a popular library that 20 million developers depend on every week. Every day brings:

• New bug reports
• Feature requests
• Security concerns
• Compatibility issues across multiple frameworks

The Math Problem:
- 100+ new issues per month
- 1,000+ open issues accumulated
- 800+ open pull requests
- 4 maintainers = impossible to keep up

Why "Just Work Harder" Fails

The critical insight here is this:

Generating code is cheap. Human attention is not.

Even the best maintainers using AI assistants still face a bottleneck:

Every solution still routes every change 
through ONE human's attention
         ↓
Human attention = the constraint
         ↓
More agents ≠ solution if human is still the bottleneck

Key Takeaway: The problem is not effort — it is architecture. You cannot solve a systems problem with individual heroics.

---

Step 2: Defining the Solution Space — The Automation Spectrum

Before building anything, you must ask: How much automation is appropriate?

The Automation Spectrum

FULL AUTOMATION          HUMAN-STEERED          MINIMAL AUTOMATION
      |__________________________|__________________________|
      
   Ship without          Human steers              Firmware in
   human reading         agent fleet               pacemakers
   any code
   
                              ↑
                         AI SDK lives
                         closer here

Why AI SDK Cannot Be Fully Automated

The Design Principle That Follows

Heavily automate the lifecycle around the human, without removing them.

The human stays in control. The factory handles everything else.

---

Step 3: Aligning Automation to Risk

Risk-Based Review Depth

Not all changes carry equal risk. The factory must recognize this:

LOW RISK                    MEDIUM RISK                 HIGH RISK
    |___________________________|___________________________|
    
Docs typo fix           Provider capability          New public API
    ↓                         ↓                           ↓
Quick glance            Focused validation           Deep review

What This Means in Practice

The factory's job is not just to generate code — it must:

1. Evaluate the fit of a change with project goals
2. Assess the risk of the change
3. Produce a documented chain of evidence so humans can calibrate their review effort

Evidence Chain Example:
  ✓ Feature confirmed missing (probe test)
  ✓ Spec fits existing architecture
  ✓ Backward compatible
  ✓ Side-effect risk: LOW
  ✓ Performance risk: NONE
  → Human reviewer: Quick approval appropriate

---

Step 4: Architecture — How the Factory Is Built

Design Principle 1: One Agent Per Task

What they tried first:

Single Agent
    ├── classify()
    ├── analyze()
    ├── implement()
    ├── review()
    └── backport()

Problem: Hard to debug, hard to test, hard to maintain

What they built instead:

Classifier Agent    → Labels issue type with confidence score
      ↓
Analysis Agent      → Investigates, writes spec, assesses fit
      ↓
Implementation Agent → Writes code, runs live tests
      ↓
Review Agent        → Scores risk, approves or flags
      ↓
Human Reviewer      → Reads evidence chain, merges
      ↓
Backport Agent      → Ports change to older versions

Why this works better:

---

Design Principle 2: Security From the Start

A public repository must assume every input is potentially hostile:

Threat Sources:
  - Malicious code in pull requests
  - Supply chain attacks
  - Resource exhaustion attempts  
  - API key exfiltration
  - Prompt injection attacks

The Defense Architecture

Layer 1: SANDBOX
┌─────────────────────────────────┐
│  Isolated Vercel Sandbox        │
│  ├── Agent code                 │
│  ├── Agent runtime              │
│  └── Only task-specific secrets │
│                                 │
│  Untrusted content can shape    │
│  proposals, but damage is       │
│  contained here                 │
└─────────────────────────────────┘
         ↓
Layer 2: NETWORK SHIELD
┌─────────────────────────────────┐
│  Controls outbound connections  │
│  Blocks paths attackers use to  │
│  exfiltrate secrets             │
└─────────────────────────────────┘
         ↓
Layer 3: HUMAN REVIEW
┌─────────────────────────────────┐
│  Nothing merges without human   │
│  approval from the SDK team     │
└─────────────────────────────────┘

---

Design Principle 3: Build Locally, Then Move to Cloud

The Progression:

Phase 1: Local CLI
  → Fast iteration
  → Easy to spot inaccuracies
  → Low overhead for experimentation

Phase 2: Managed Cloud Infrastructure
  → GitHub webhooks trigger automatically
  → Queue processes issues at scale
  → Monitoring UI tracks parallel runs

Infrastructure Stack:

---

Step 5: Tracing a Real Change Through the Factory

Let's follow Issue #17898 — a request for blocked-domain support in OpenAI web search.

Stage 1: Classification

Input: Community issue text
         ↓
Classifier Agent runs
         ↓
Output: 
  Type: Feature Request
  Confidence: HIGH
  Rationale: [documented in comment]
  Label: applied to issue

Stage 2: Analysis

Classifier output + codebase context
         ↓
Analysis Agent runs probe:
  issue-17898-type-probe.ts
         ↓
Probe FAILS → confirms feature is missing on main
         ↓
Agent builds spec:
  - Add optional blockedDomains filter
  - Map to provider's blocked_domains field
  - Fits provider-adapter architecture ✓
  - Backward compatible ✓
  - Documentation changes scoped ✓

Stage 3: Implementation

Spec from Analysis Agent
         ↓
Implementation Agent:
  - Writes the code change
  - Runs live end-to-end test
    (OpenAI search with wikipedia.org blocked)
  - Confirms domain is unreachable ✓
  - Opens Pull Request with evidence attached

Stage 4: Automated Review

Pull Request
         ↓
Review Agent scores the change:
  Implementation completeness: FULL ✓
  Side-effect risk: LOW ✓
  Performance risk: NONE ✓
  Backwards-compatibility risk: LOW ✓
         ↓
Agent approves PR

Stage 5: Human Review

Human (Lars) reads:
  - Classification rationale
  - Analysis spec and probe results
  - Implementation evidence
  - Review scores
         ↓
Reviews actual code diff
         ↓
Merges PR #18033 to main

Stage 6: Backporting

Merge detected
         ↓
Backport Agent opens:
  PR #18035 → v6 branch (clean apply)
  PR #18036 → v5 branch (conflict detected)
         ↓
For v5 conflict:
  Agent labels conflicted state
  Identifies fix
  Validates fix
  Pushes resolution (17 minutes later)
         ↓
Human reviews and merges both

---

Step 6: Results and What They Mean

Measured Outcomes After 4 Weeks

PRs authored by factory:    25–35% of weekly merges
Backports (v6):             >50% of weekly merges  
Issues closed by factory:   70–80% in July
Open issues:                1,022 → 844 (↓18%)
Open bugs:                  ↓25%

Why These Numbers Matter

The factory did not replace human judgment — it multiplied human capacity:

Before factory:
  Human attention → bottleneck → backlog grows

After factory:
  Factory handles: classification, analysis, 
                   implementation, review, backports
  Human handles:  final judgment on evidence chain
  
Result: Same humans, dramatically more throughput

---

Step 7: The Feedback Loop — How the Factory Improves

Every factory run ends in one of four states:

The Key Insight

Traditional engineering job:
  Write code → test code → ship code

Agentic engineering job:
  Improve the factory → factory writes, tests, ships code

Every failure is signal. Every signal expands the automation boundary. The factory gets better every week.

---

Summary: Core Principles to Remember

1. SCALE THE SYSTEM, NOT THE HUMANS
   Human attention is the constraint — design around it

2. MATCH AUTOMATION DEPTH TO RISK
   Not everything needs the same level of scrutiny

3. ONE AGENT PER TASK
   Focused, testable, debuggable, improvable

4. SECURITY IS NOT OPTIONAL
   Public repos face real attackers — sandbox everything

5. BUILD INCREMENTALLY
   Local CLI first, cloud infrastructure second

6. EVIDENCE CHAINS ENABLE TRUST
   Humans can only review efficiently if agents document their reasoning

7. FAILURES ARE FEEDBACK
   Every broken run makes the factory smarter

---

Quick Self-Check Questions

1. Why does adding more individual agents not solve the maintainer bottleneck problem?
2. Where on the automation spectrum should foundational infrastructure sit, and why?
3. What are the three layers of security defense in the factory?
4. Why is "one agent per task" better than one agent with many skills?
5. What happens when a factory run is marked "flawed" — and why is that valuable?
6. How does the evidence chain change the nature of human review?