Teaching Sidekick to say no: automated data curation with LLM judge consensus (2026) - Shopify

Concept 1: The Core Problem with Training Data

What it is

When you train an AI model, the quality and type of data matters more than the algorithm or computing power.

The key insight

"Training data has always been the hard part of machine learning. Not the architecture, not the compute, but the data itself."

Why this matters for LLMs specifically

Before LLMs, the challenge was labeling and feature engineering. With LLMs, the challenge shifts to teaching behavior:

The blind spot

Production logs (real user interactions) only capture successes. Failed or impossible requests never make it into training data. So the model learns from a biased sample of reality.

---

Concept 2: The Production Data Blind Spot

What it is

A structural gap in training data caused by how production systems work.

The mechanism

User makes request
       ↓
Model handles it successfully
       ↓
Gets evaluated and logged ✓
       ↓
Enters training corpus

But what about this path?

User makes impossible request
       ↓
Model should refuse... but doesn't know how
       ↓
Model improvises badly
       ↓
Never logged as a "refusal example"
       ↓
Never enters training corpus ✗

The consequence

The model is trained entirely on successful queries. When it encounters something impossible, it has no learned behavior to fall back on. It improvises — usually badly.

Real example from Sidekick

A merchant asks: "Find me customers who are doctors."

Shopify doesn't store occupation data. The correct answer is: "I can't do that."

Instead, the model generated a query that returned zero results — giving the merchant the false impression that no doctors exist in their customer base, rather than that the request was impossible.

---

Concept 3: What Sidekick Is and How It Works

Architecture overview

Sidekick is Shopify's AI assistant for merchants. It has two layers:

Merchant Request: "Send a discount to my best customers"
                        ↓
              ┌─────────────────┐
              │  Outer Planner  │  ← Interprets overall intent
              └────────┬────────┘
                       │ Routes to...
          ┌────────────┼────────────┐
          ↓            ↓            ↓
   Segmentation    Analytics      Email
     Skill          Skill         Skill

Why this matters

Each skill model is specialized. The segmentation skill, for example, converts plain English into Shopify's domain-specific query syntax — a language most merchants can't write themselves.

The specific problem

The segmentation skill was fine-tuned on production data — all successful queries. It had zero refusal examples. So it never learned to say "I can't do that."

---

Concept 4: Why Naive Solutions Fail (The Merge Problem)

The first attempt

Shopify partnered with Toloka (a data annotation team) to create ~600 refusal examples. Then they tried the obvious fix:

Production data (successes) + Toloka data (refusals) = Merged dataset → Fine-tune

Why it failed: Contradictory Labels

The same query appeared in both datasets with opposite labels:

Production data says:  "Find customers who bought last month" → ✓ VALID
Toloka data says:      "Find customers who bought last month" → ✗ REFUSE

The model received contradictory training signals for semantically similar inputs. This caused:

• Degraded stability
• Worse generation quality
• No reliable improvement in refusal behavior

The lesson

You can't just merge datasets with conflicting labels. You need a way to resolve conflicts at scale — which manual annotation can't do cheaply.

---

Concept 5: LLM Judges as Automated Data Curators

The core idea

Instead of hiring more human annotators, use frontier LLMs as automated judges to evaluate and label training data.

How it works

Small high-quality Toloka dataset (seed)
              ↓
    Calibrate 4 LLM judges using seed
              ↓
    Run judges across full production corpus
              ↓
    Judges label AND resolve conflicts

Why use multiple judges?

A single LLM judge can be wrong or inconsistent. Using four independent judges creates a panel — similar to peer review in science.

Calibration step (critical)

Before judging anything, each LLM is shown few-shot examples from the Toloka dataset:

"Here's a query. Here's what a human expert said about it. Now use this as your reference."

This anchors the judges to human ground truth, not the model's own assumptions about what Shopify can or can't do.

---

Concept 6: Strict Consensus Over Loose Confidence

The rule

A label is only accepted if all four judges agree on:

1. The decision (valid vs. refuse)
2. The reasoning behind it

Judge A: REFUSE (because Shopify doesn't store occupation data)
Judge B: REFUSE (because Shopify doesn't store occupation data)
Judge C: REFUSE (because Shopify doesn't store occupation data)
Judge D: REFUSE (because Shopify doesn't store occupation data)
                        ↓
              ✓ CONSENSUS REACHED → Label accepted

Judge A: REFUSE
Judge B: REFUSE
Judge C: VALID
Judge D: REFUSE
              ↓
         ✗ NO CONSENSUS → Example discarded (sent to human review)

Precision vs. Recall tradeoff

"Better to miss a valid refusal than to mislabel a legitimate merchant request."

Why this matters

Noisy labels in training data are worse than missing data. A wrong label actively teaches the model bad behavior. A missing example just means less coverage.

---

Concept 7: A Mutually Exclusive Taxonomy

The problem with vague categories

If categories overlap, judges will disagree — not because the task is hard, but because the categories themselves are ambiguous.

The solution: Four clean, non-overlapping categories

Why mutual exclusivity is non-negotiable

Overlapping categories → Judge disagreements
                              ↓
                    Inconsistent labels
                              ↓
                    Unstable fine-tuning
                              ↓
                    Worse model behavior

Clean categories → faster calibration, more consistent labels, more stable training.

---

Concept 8: The Data Flywheel

What it is

A self-reinforcing feedback loop where each model deployment generates data that improves the next version.

The loop

┌─────────────────────────────────────────────────────┐
│                                                     │
│  1. Improved model ships to production              │
│                    ↓                                │
│  2. Real merchant queries come in (including        │
│     new edge cases, new phrasings)                  │
│                    ↓                                │
│  3. Judge ensemble labels new queries               │
│     (discards disagreements)                        │
│                    ↓                                │
│  4. Accepted labels added to training corpus        │
│                    ↓                                │
│  5. Next fine-tuning run starts with MORE           │
│     data and CLEANER labels                         │
│                    ↓                                │
│  6. Better model ships → back to step 1 ──────────►│
└─────────────────────────────────────────────────────┘

Why this is powerful

• Merchants constantly find new ways to phrase the same request
• Partial phrasings expose gaps not in the original training set
• Each deployment funds the next improvement automatically
• No manual annotation needed at scale

The key requirement

Production traffic cannot just sit in dashboards. It must become the input to the next training run.

---

Concept 9: Measuring Success

The results

What Cohen's Kappa means

It measures inter-rater reliability — how much agreement exists beyond random chance:

---

Concept 10: Refusals as Product Features

The mindset shift

Most teams think of refusals as failures — the model couldn't do something.

Shopify reframes this: a truthful refusal is a product feature.

Why hallucination is worse than refusal

The design implication

"We spent a lot of time teaching the model when to say yes. We should have started with when to say no."

A refusal, ideally paired with a suggestion ("I can't find customers by occupation, but I can segment by purchase history"), keeps the user experience intact.

---

Summary: The Full System at a Glance

PROBLEM: Production data has no refusal examples
              ↓
SEED: Small high-quality human annotations (Toloka, ~600 examples)
              ↓
JUDGES: 4 frontier LLMs calibrated on seed data
              ↓
TAXONOMY: 4 mutually exclusive categories (no ambiguity)
              ↓
CONSENSUS GATE: All 4 judges must agree on decision + reasoning
              ↓
CORPUS: Cleaned, conflict-resolved training data
              ↓
FINE-TUNING: Model learns to refuse impossible requests
              ↓
DEPLOYMENT: Better model ships
              ↓
FLYWHEEL: New production traffic → new labels → next training run
              ↓
              (loop continues, improving each cycle)

---

Four Key Lessons to Remember