After studying this material, you should be able to:

1. Explain what an automated feedback loop is and why it matters for AI systems
2. Describe the three-tier evaluation framework and what each tier measures
3. Understand how convergence and damping work in self-improving systems
4. Identify the prerequisites required before implementing this architecture
5. Apply the core pattern to a new domain

---

Step-by-Step Teaching

Step 1: The Core Problem — Why AI Systems Stop Getting Better

Imagine a team where an expert writes a brilliant code review comment. That comment helps one person. Then it disappears forever.

The next contributor makes the exact same mistake. Another expert writes the exact same comment. This repeats indefinitely.

Contributor A → Makes mistake → Expert reviews → Wisdom evaporates
Contributor B → Makes same mistake → Expert reviews → Wisdom evaporates
Contributor C → Makes same mistake → Expert reviews → Wisdom evaporates

The key insight from the article:

Repeated review comments are not really comments anymore — they are undocumented requirements

When the same guidance appears frequently enough, it no longer belongs in a human review. It belongs inside the system itself.

Why this is architectural, not a tooling problem:

• The signal (expert feedback) already exists
• Nobody is collecting it systematically
• Nobody is feeding it back into the AI that produced the artifact
• Result: organizations generate enormous insight and retain almost none of it

---

Step 2: The Solution Pattern — Three Core Moves

The article describes a repeatable architecture built on three moves. Think of it as a cycle:

┌─────────────────────────────────────────────────────┐
│                                                     │
│   GENERATE → REVIEW → COLLECT SIGNAL → IMPROVE     │
│       ↑                                    │        │
│       └────────────────────────────────────┘        │
│                                                     │
└─────────────────────────────────────────────────────┘

Move 1: Build a generator Create an AI that produces structured, verifiable output. In the article's example, this was creating-sf-skill — a meta-skill that generates complete skill directories with consistent formatting, naming conventions, and file structure.

Move 2: Measure quality with a layered evaluation framework You need to prove the system is improving before you can improve it.

Move 3: Feed review signal back automatically Close the loop by mining human feedback and updating the generator's instructions.

---

Step 3: The Three-Tier Evaluation Framework

This is the measurement engine. Each tier catches failures the others cannot.

Tier 1 — Activation Testing (Does it work in the real world?)

Why this matters first: If the output doesn't function correctly in a crowded real environment, nothing else matters. You cannot skip to quality if the basics fail.

Tier 2 — Deterministic Validation (Does it follow the rules?)

Key principle: Use scripts for everything that has an objectively correct answer. This is pure determinism — no judgment required.

Tier 3 — Semantic Quality (Is it actually good?)

The holistic principle the article establishes:

Deterministic validation → measures what is objectively verifiable
LLM judgment           → measures what requires context and understanding
Neither replaces the other — they answer different questions

---

Step 4: The Feedback Loop Mechanism

Now you can measure quality. But how do you actually improve the generator automatically?

The Weekly Automated Cycle

Gate conditions (must both be true before running):

• At least 15 merged PRs reviewed in the window
• No improvement PR already open

Why 15 PRs minimum? Fewer than that risks overfitting to one reviewer's personal style rather than community consensus.

What happens when the loop runs:

Step 1: SCAN
        Collect all review comments from merged PRs in the window

Step 2: SYNTHESIZE  
        Identify patterns — which comments appeared repeatedly?
        (A comment appearing once = personal preference)
        (A comment appearing 5+ times = undocumented requirement)

Step 3: ISOLATE GAPS
        Separate patterns into: structural gaps vs. semantic gaps

Step 4: UPDATE
        Rewrite the generator's instructions to encode the pattern

Step 5: VALIDATE
        Run the three-tier framework on new output
        Only merge if all tiers pass

The Efficacy Metric

How do you know the loop is working?

Track frequency of repeated comment patterns across cycles:

Cycle 1: Pattern X appears in 5 PRs → Loop encodes it
Cycle 2: Pattern X appears in 0 PRs → Loop worked ✓

The decline in repeat comments is the measurement of success.

---

Step 5: Convergence and Damping — Why the System Stabilizes

This is the most counterintuitive part. Most people assume self-improving systems mutate forever. This one does not.

The Natural Damping Mechanism

Generator improves
    → Produces fewer mistakes
        → Reviewers write fewer comments
            → Less signal per cycle
                → Smaller changes applied
                    → System stabilizes

This is not a bug. It is the designed behavior.

Evidence from the Six Cycles

The system literally ran out of things to fix. That is what success looks like.

What Happens When Standards Change

Convergence is conditional, not permanent.

Standards change
    → Generator's instructions become stale
        → Reviewers flag the same new issue repeatedly
            → Frequency spike crosses threshold
                → Loop restarts automatically
                    → System re-converges at new equilibrium

The article demonstrates this with a real example: frontmatter conventions changed, 5 PRs flagged the same stale guidance in one window, the loop corrected itself without human intervention.

---

Step 6: Applying the Pattern to a New Domain

The architecture is domain-agnostic. The article demonstrates this by applying it to an automated code reviewer bot.

Same three moves, different signal source:

The pattern held. First cycle produced four improvements to the bot's reference files and severity calibration.

---

Step 7: Prerequisites — When This Architecture Works (and When It Doesn't)

Before implementing this, verify all three conditions hold:

✅ Condition 1: Minimum Signal Threshold

• You need 15+ reviewed artifacts within your cycle window
• If only 2 contributors touch output quarterly, the loop overfits to personal style
• Rule of thumb: If you cannot collect 15+ reviewed artifacts per cycle, you will produce noise, not signal

✅ Condition 2: Structured, Verifiable Output

• Your generator must produce output with both:
  • A structural contract (schema, naming conventions, file layout) → enables Tier 2
  • Semantic requirements (quality, clarity, correctness) → enables Tier 3
• Free-form prose with no structural contract collapses Tier 2 entirely

❌ When the pattern breaks down:

• Rubber-stamp review culture — approvals without real feedback contain no signal to mine
• Purely subjective output — if "good" is entirely taste-driven with no measurable proxies, convergence dissolves

---

Summary: The Complete Mental Model

PREREQUISITE CHECK
├── Enough review volume? (15+ per cycle)
├── Structured output? (mechanical + semantic dimensions)
└── Real review culture? (genuine feedback, not rubber stamps)

IF YES → BUILD THE LOOP

GENERATOR produces structured artifacts
    ↓
REVIEWERS provide feedback (signal)
    ↓
LOOP (weekly, gated):
    Scan → Synthesize → Isolate gaps → Update → Validate
    ↓
THREE-TIER EVALUATION confirms improvement:
    Tier 1: Activation (does it work?)
    Tier 2: Deterministic (does it follow rules?)
    Tier 3: Semantic (is it actually good?)
    ↓
CONVERGENCE: fewer comments → smaller changes → stability
    ↓
RESTART when standards shift (frequency spike = new signal)

The end state: Reviewer wisdom stops evaporating into forgotten comments. It accumulates automatically into the system that produces the next artifact. Every participant in the review process makes the system better — without anyone doing extra work.