2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify

TL;DR

Shopify built an AI-powered simulation system that lets any online store — even tiny ones with zero visitors — run A/B tests in minutes instead of weeks, using hundreds of robot shoppers guided by a massive AI brain, and then had to reinvent how AI inference works to make it affordable at scale.

---

Key Discoveries (13 total)

1. SimGym: A/B testing without real traffic — Shopify built a system that sends hundreds of AI-powered "robot shoppers" (each with a unique persona, budget, and shopping intent) through a store's website in a controlled environment, simulating what real customers would do. This lets even tiny stores with zero real visitors test layout changes and get results in minutes instead of waiting weeks for enough real shoppers to show up.

2. Inference latency directly equals money — Because the cloud browsers (up to 2,000 running simultaneously) bill by the minute, every second the AI spends "thinking" costs real money. A ~20% reduction in AI response time cut the cost per merchant test run by ~10% and increased how many tests could run per day by ~12%.

3. Open-source AI made the economics work — Switching from paid API models (like GPT-4o/GPT-5) to a self-hosted open-source 120-billion-parameter model (gpt-oss-120b) dropped session times from 15+ minutes to under 3 minutes and replaced unpredictable per-token costs with fixed, predictable infrastructure costs.

4. Standard AI serving tools weren't built for this — Typical AI chat systems handle short, independent requests arriving randomly. SimGym fires 600 correlated, long requests all at once, where each step depends on the previous one, and each session grows to 89,000–127,000 tokens. This broke the assumptions of standard serving software (vLLM), requiring custom engineering.

5. NVIDIA's newest "Blackwell" chips gave a 5.2x speed boost — When the team benchmarked the same AI workload on different hardware, NVIDIA's latest B200 GPU ("Blackwell") produced 57,000 tokens per second versus 11,000 on the previous H200 generation — a 5.2x speedup that made Blackwell the clear hardware choice.

6. Speculative decoding works especially well for robot shoppers — Speculative decoding is a technique where a small "draft" AI quickly guesses several upcoming words, and the big AI verifies them all at once instead of one-by-one. Because SimGym's robots always output predictable structured data (like "action", "nodeSelector"), the draft model's guesses are accepted at a high rate, making this technique particularly effective here.

7. A custom AI attention kernel was built and released publicly — NVIDIA's kernel engineers used SimGym's real production data to build a custom optimized piece of low-level GPU code (a "kernel") specifically for long-context speculative decoding on Blackwell hardware. This delivered roughly a 2x speedup for that operation and was released publicly as FlashInfer PR #2265 (available in FlashInfer 0.6.1).

8. Three advanced techniques were made to work together simultaneously — Getting async scheduling (for throughput), guided decoding (to enforce valid JSON output), and speculative decoding (for speed) to all work at the same time without canceling each other out was a significant engineering challenge, resulting in a public contribution to vLLM (PR #29821).

9. Splitting each GPU into two virtual instances cut latency by 20% — Using a technique called MIG (Multi-Instance GPU) partitioning — essentially dividing each physical GPU into two independent slices — reduced average AI response time from 27.8 to 21.9 seconds, shortened total session duration from 7.3 to 6.6 minutes, and increased daily merchant test runs from 1,311 to 1,463, with no drop in quality.

10. Smarter prompt structure unlocked a 12% throughput boost — By reorganizing how instructions are written to the AI (moving the parts that change per-user out of the shared beginning of the prompt), the system could reuse cached computations more often. This alone produced a ~12% throughput increase at high concurrency (1,000+ simultaneous sessions).

11. Reducing the AI's "thinking effort" was a failed shortcut — The team tried telling the AI to reason less carefully to save time. Session duration dropped ~75%, but error rates jumped from under 1% to as high as 10.9% — making the outputs unreliable and the tradeoff not worth it.

12. The system now runs 400,000 simulated shopping sessions per day — From a prototype making simple API calls seven months ago, SimGym scaled to a dedicated cluster of 48 NVIDIA B200 GPUs running hundreds of thousands of sessions daily, with cost per merchant run now in single digits and still falling.

13. NVIDIA published a specialized AI "draft model" for this use case — To support speculative decoding for SimGym's specific model, NVIDIA trained and publicly released a dedicated EAGLE-3 speculative decoding head (nvidia/gpt-oss-120b-Eagle3-throughput on HuggingFace), making this optimization available to anyone using the same base model.

---

Why This Matters

For small businesses, this is a genuine equalizer. A/B testing — comparing two versions of a website to see which sells better — has always required lots of real visitors to get statistically reliable results. Small merchants simply don't have that traffic, so they've been flying blind when making design decisions. SimGym gives them the same competitive intelligence that only large retailers could previously afford.

For the AI industry, this project is a case study in what happens when you push AI systems far beyond their original design limits. The engineering solutions developed here — custom GPU kernels, new ways to combine optimization techniques, hardware benchmarking with real production data — didn't stay inside Shopify. They were contributed back to open-source projects (vLLM, FlashInfer) and published publicly, meaning every team building AI-powered applications benefits.

For anyone building "agentic AI" (AI that takes sequences of actions to complete a task, rather than just answering a single question), this article is a roadmap. The specific challenges of long context, structured outputs, bursty correlated traffic, and sequential dependencies are problems every agentic system will eventually face — and this team documented both the problems and the solutions in detail.

The Research Approach

The team at Shopify didn't set out to do academic research — they were solving a real operational problem: how do you run A/B tests for merchants who don't have enough real website visitors to get reliable results? Their approach was fundamentally engineering-driven: build something, measure it, find the bottleneck, fix it, repeat.

The methodology had three distinct phases:

Phase 1 — Prove the concept works. Use existing commercial AI tools (GPT-4o, then GPT-5) to see if robot shoppers could behave realistically enough to be useful. They could: 75 out of 100 simulated buyers completed their shopping task. This was enough to justify going further.

Phase 2 — Make it economically viable. Once the concept worked, the per-token costs of commercial APIs became unsustainable at the scale they needed (hundreds of thousands of sessions per day). They switched to a self-hosted open-source model, which converted unpredictable variable costs into fixed infrastructure costs — like the difference between paying per taxi ride versus buying a company car.

Phase 3 — Make the infrastructure fast enough. Self-hosting the model revealed that standard AI serving software wasn't designed for their specific workload. This triggered a deep collaboration between Shopify's application engineers, NVIDIA/CentML's infrastructure engineers, and GPU kernel specialists — each contributing their layer of expertise.

The key methodological principle throughout: measure with real production data, not synthetic benchmarks. Every optimization decision was driven by actual traffic patterns from SimGym's live workload.

---

How It Works in Practice

The Shopping Loop

Each simulated shopping session works like this:

1. A cloud browser (running a real Chromium browser, the same engine behind Google Chrome) opens the merchant's store
2. The browser sends the current page state — a structured description of everything visible on the page — to the AI model
3. The AI reads a short description of who this shopper is (their persona, budget, and what they're looking for), its memory of what it has already done, and the page description (between 3,000 and 15,000 words worth of data)
4. The AI returns a single instruction in JSON format — a structured data format that looks like {"action": "click", "nodeSelector": "#add-to-cart-button"} — telling the browser exactly what to do next
5. The browser executes that action and sends back the new page state
6. This loop repeats 10–13 times per session

Up to 600 of these loops run simultaneously, all firing at the same moment when a test begins.

The Hardware Benchmarking Process

Rather than guessing which hardware would be fastest, the NVIDIA team took Shopify's actual production data — the real distribution of prompt lengths, how many sessions run at once, how tokens accumulate over a session — and replayed that exact workload against three different GPU generations (H100, H200, and B200) in a controlled comparison. Think of it like a car manufacturer testing the same route on three different engines rather than relying on the manufacturer's spec sheet.

The result was concrete and decisive: 11,000 tokens per second on H200 versus 57,000 on B200 — a 5.2x difference that made the hardware choice straightforward.

The MIG Partitioning Experiment

MIG (Multi-Instance GPU) partitioning means taking one physical GPU and dividing it into two completely independent virtual GPUs, each with its own dedicated memory. The team ran a controlled end-to-end experiment at real production traffic levels, comparing the same workload on whole GPUs versus split GPUs. The split configuration reduced average AI response time from 27.8 to 21.9 seconds — because each smaller instance handles fewer simultaneous requests and therefore queues less — and increased daily merchant test runs from 1,311 to 1,463, with no measurable drop in output quality.

The Failed Shortcut

The team tried an intuitive optimization: tell the AI to "think less carefully" to save time. Session duration dropped dramatically — about 75% faster. But error rates jumped from under 1% to as high as 10.9%. This is a useful case study in how speed and reliability trade off: the faster sessions were producing unreliable outputs, which would have made the entire system untrustworthy. The experiment was abandoned.

---

What Makes This Different

The Core Difference: Agentic vs. Conversational AI

Standard AI serving software was designed for conversational AI — think of a customer service chatbot. Each question is independent. Requests arrive randomly throughout the day. Responses are short. The system can handle one question, forget it, and move on.

SimGym is agentic AI — the AI takes a sequence of actions to complete a task, where each action depends on what happened before. This creates fundamentally different engineering requirements:

None of the standard tools were built for the right column of that table.

Why Speculative Decoding Works Especially Well Here

Speculative decoding — where a small "draft" AI quickly guesses several upcoming words, and the large AI verifies them all at once — is a general technique, but it works particularly well for SimGym for a specific reason. The AI's outputs are always structured JSON with predictable field names. When the output always contains words like "action", "nodeSelector", and "method", the small draft model can guess those words correctly almost every time. High acceptance rates mean the large model spends more time confirming correct guesses and less time correcting wrong ones — which is where the speed gain comes from.

The Prompt Caching Insight

Modern AI systems can cache (save and reuse) the computation for parts of a prompt that don't change between requests. The team discovered that their original prompt structure had the variable parts (the shopper's persona and intent) mixed into the beginning of the prompt, which prevented caching from working. By moving those variable parts to the end of the prompt — keeping the stable instructions at the beginning — the system could reuse cached computations far more often. This structural change alone produced a 12% throughput increase at high concurrency, with no other changes to the AI or hardware.

---

Overcoming Challenges

Challenge 1: The Model Is Memory-Bound, Not Compute-Bound

The gpt-oss-120b model is a "Mixture-of-Experts" (MoE) architecture — imagine a committee of 120 specialists where only a few are consulted for any given question, but all 120 must be present in the room. Only a fraction of the model's 120 billion parameters activate for each token, making the actual computation comparable to a much smaller model. But all 120 billion parameters must sit in GPU memory simultaneously.

This means the bottleneck isn't the GPU's calculation speed — it's how fast data can be moved from memory to the processor. Standard serving software (vLLM) was optimized for compute-bound workloads and wasn't efficient here.

Solution: The team applied several techniques specifically targeting memory movement: MXFP4 quantization (storing expert weights in a compressed format that takes up less memory space, like compressing a photo file without losing visible quality), FP8 attention for the KV-cache (a similar compression for the memory that stores context), and custom-built GPU kernels (low-level code that controls exactly how data moves through the hardware) optimized for this specific model architecture.

Challenge 2: Three Techniques That Interfered With Each Other

The team needed three optimizations running simultaneously:

• Async scheduling: processes multiple requests in overlapping batches for higher throughput
• Guided decoding: forces the AI to output valid JSON by constraining which tokens it can produce at each step
• Speculative decoding: uses a draft model to propose multiple tokens for batch verification

The problem: each technique made assumptions about how the others worked, and those assumptions conflicted. Enabling all three together caused them to partially cancel each other out.

Solution: Engineers from Shopify, NVIDIA, and the vLLM open-source project worked through the conflicts systematically, modifying the underlying vLLM software to allow all three to coexist. The fix was contributed back to the public vLLM codebase (PR #29821), meaning any team facing the same combination of requirements can now use it.

Challenge 3: No Existing Benchmarks Captured Their Workload

Standard AI performance benchmarks simulate random, independent, short requests — the chatbot scenario. Running those benchmarks on SimGym's hardware would have given misleading results, because SimGym's actual workload (600 simultaneous correlated long-context requests) behaves completely differently.

Solution: The NVIDIA team collected Shopify's actual production traffic data — real prompt lengths, real concurrency patterns, real token accumulation curves — and built a custom benchmark that replayed that exact workload. This meant every hardware and software decision was validated against the real problem, not a simplified approximation of it.

Challenge 4: MIG Partitioning Conflicts With Other Optimizations

Splitting each GPU into two virtual instances (MIG) gave a 20% latency reduction, but it created a constraint: each virtual instance has less memory than the full GPU. This meant the draft model used for speculative decoding no longer fit alongside the main model in the same instance. The two optimizations are mutually exclusive on the same hardware.

Solution: Rather than treating this as a failure, the team recognized these as two different optimization profiles suited to different configurations. MIG partitioning is being deployed first because it delivers immediate gains with no quality risk. Speculative decoding will be deployed on full-GPU configurations where memory allows. The sequencing of which optimization to deploy when became its own engineering decision, driven by the specific constraints of each approach.