Step-by-Step Teaching

Step 1: The Core Problem — Why Subjective Categories Are Difficult

The Standard Assumption in Machine Learning

Traditional ML assumes:

Human has clear definition → Labels examples → Model learns

This works perfectly for objective categories:

Where This Breaks Down

Now consider subjective categories:

• "Healthy food"
• "Inspiring artwork"
• "Inappropriate content"

These categories share three critical problems:

Problem 1 — Shifting Boundaries What counts as "healthy" changes with culture, personal philosophy, and evolving science

Problem 2 — Incomplete Mental Models People don't know their own full definition upfront. They discover edge cases as they encounter them

Problem 3 — Concept Evolution Traditional workflows force rigid labels even when the human's own understanding is still developing

Quick Check ✓

Ask yourself: Could two reasonable people disagree about whether a fruit smoothie is "healthy food"? If yes — you've identified a subjective gray area.

---

Step 2: The Traditional Approaches and Why They Fall Short

Before understanding Agile Deliberation, you need to understand what it replaces:

Approach 1: Zero-Shot Baseline

Give AI a simple text prompt → AI classifies immediately

❌ Uses generic AI assumptions, not YOUR specific intent

Approach 2: Manual Prompt Engineering

Human writes prompt → Tests it → Manually rewrites prompt → Repeat

❌ Two major failure modes identified in research:

• Cognitive fatigue — Trying to predict every possible edge case is mentally exhausting
• Anxiety — Fixing one mistake might break something already working

Approach 3: Traditional Active Learning

AI selects uncertain examples → Human labels them → Model improves

❌ Optimizes for machine uncertainty, not human conceptual clarity

---

Step 3: Introducing Agile Deliberation — The Core Idea

The Paradigm Shift

The Central Metaphor

Think of it like drafting a legal contract with a smart assistant:

• You don't need to know every clause upfront
• The assistant surfaces scenarios you hadn't considered
• You react and clarify
• The contract gets sharper with each round

---

Step 4: Stage 1 — Concept Scoping

What Happens Here

The system guides you to decompose your broad concept into a structured hierarchy before any images are labeled.

The Structure

Broad Concept: "Healthy Food"
        │
        ├── POSITIVE sub-categories (what TO include)
        │       ├── Salads
        │       ├── Grain Bowls
        │       └── Lean Proteins
        │
        └── EXPLICIT EXCLUSIONS (what NOT to include)
                ├── Processed Foods
                └── Raw, Unprepared Ingredients

Why This Matters

Without scoping, your AI has no structured baseline — it's working from a single vague label. Scoping creates an instructional blueprint that grounds the model's initial understanding.

Key Terminology

---

Step 5: Stage 2 — Concept Iteration

The Active Testing Phase

Once the initial structure exists, the system moves from planning to testing.

How It Works — The Loop

Step 1: VLM mines semantically BORDERLINE images
            ↓
Step 2: System presents curated gallery to human
            ↓
Step 3: Human provides natural feedback
        ("Yes this fits" / "No, exclude this")
            ↓
Step 4: Automatic prompt optimization loop updates model
            ↓
Step 5: Repeat with new borderline images

The Fruit Smoothie Example

The system surfaces an image of a fruit smoothie and forces a decision:

Is this HEALTHY FOOD?

Option A: YES — contains raw fruit, natural ingredients
Option B: NO — high sugar content, processed form

Your answer reveals YOUR hidden assumption
→ System updates the classifier accordingly

Why Borderline Images Are Powerful

"Confronting ambiguous edge cases drives users to reflect on their own hidden assumptions, clarify their true intent, and update the policy rules accordingly."

This is the key insight: you don't know your own full definition until you're challenged by hard cases.

Contrast With Traditional Active Learning

---

Step 6: The Evidence — What the Research Found

Study Design

• 18 live sessions, each 1.5 hours
• 2 subjective domains tested: "Pay-to-Play" content and "Healthy Food"
• Participants tested Agile Deliberation against multiple baselines in randomized order

Finding 1: Better Classifier Performance

Performance Improvement:
─────────────────────────────────────────
Agile Deliberation    ████████████████  HIGH
Zero-shot baseline    ████████          MEDIUM  
Manual deliberation   ██████            LOW (sometimes degrades!)
─────────────────────────────────────────

Gains were most dramatic on the hardest subjective images — exactly where traditional classifiers fail

Finding 2: Reduced Cognitive Load

Participants reported statistically significant reductions in:

100% of participants preferred Agile Deliberation over manual methods

Finding 3: Diagnostic Value

The system acted as a "blind spot detector" — surfacing scenarios experts hadn't anticipated, making it a tool for discovering your own definition, not just implementing it.

---

Step 7: Synthesizing the Big Picture

The Complete Framework at a Glance

FUZZY IDEA
    │
    ▼
[CONCEPT SCOPING]
Define sub-categories + explicit exclusions
    │
    ▼
[CONCEPT ITERATION]
VLM surfaces borderline images
    → Human gives natural feedback
    → Automatic prompt optimization
    → Repeat until sharp
    │
    ▼
PRECISE, ALIGNED CLASSIFIER

Why This Matters Beyond the Lab

Agile Deliberation has real-world applications wherever subjective judgment meets scale:

---

Summary: The 5 Core Concepts to Remember

---

Self-Assessment Questions

1. Why does traditional ML labeling fail for subjective categories?
2. What is the difference between in-scope and out-of-scope definitions?
3. How does Agile Deliberation's use of borderline images differ from traditional active learning?
4. What two cognitive problems does manual prompt engineering cause?
5. Describe a real-world scenario where you would apply Agile Deliberation and walk through how the two stages would work.