After studying this material, you should be able to:

1. Explain why standard A/B testing fails in two-sided advertising marketplaces
2. Define SUTVA and describe how it gets violated through budget cannibalization
3. Describe how Budget Split creates isolated experimental conditions
4. Explain how Orthogonal Concurrency enables multiple simultaneous experiments
5. Identify residual sources of interference that persist even with these solutions

---

Step-by-Step Study Material

Step 1: Understanding the Foundation — What is a Two-Sided Marketplace?

Before anything else, you need to understand what makes this environment special.

A standard app has one side:

Company → Users

A retail media marketplace has two sides:

Advertisers (pay money) → Platform → Customers (see ads)

Key constraint: Advertisers have finite budgets. This finite resource is what creates all the problems we will discuss.

Think of it like this: Imagine two restaurants sharing one kitchen. If Restaurant A uses all the stove burners, Restaurant B cannot cook. They are not independent.

---

Step 2: Why Standard A/B Testing Breaks Here

The Core Assumption: SUTVA

Every standard A/B test relies on SUTVA — Stable Unit Treatment Value Assumption.

SUTVA has one critical requirement:

What happens to User A must NOT affect what happens to User B

In a normal app, this holds reasonably well:

• User A sees a blue button → does not affect User B seeing a red button
• They are independent

How SUTVA Gets Violated in AdTech

Now introduce a shared budget and watch independence collapse:

Scenario:
- Variant A = New, more efficient bidding algorithm
- Variant B = Old baseline algorithm
- Both draw from the SAME campaign budget pool

The chain reaction:

Variant A wins more auctions
        ↓
Budget depletes faster
        ↓
Variant B has less money to spend
        ↓
Variant B performs worse artificially
        ↓
You conclude Variant A is better...
        ↓
...but was it really better, or did it just starve Variant B?

This is called Cannibalization Bias — one variant actively degrades the other's performance through shared resource consumption.

Analogy: You test two athletes on a treadmill, but they share one oxygen tank. The stronger athlete breathes more, leaving the weaker one gasping. You cannot fairly compare their natural abilities.

---

Step 3: The Solution — Budget Split

The Core Idea

Instead of one shared budget pool, create separate, isolated budget pools per variant.

BEFORE Budget Split:
Campaign Budget ($1000)
    ├── Variant A draws from here
    └── Variant B draws from here (competing!)

AFTER Budget Split:
Campaign Budget ($1000)
    ├── Sub-budget A ($500) → Only Variant A draws from this
    └── Sub-budget B ($500) → Only Variant B draws from this

How It Works — Three Mechanical Steps

Step 1: Traffic Randomization

• Users are assigned to either Treatment or Control group
• This is standard A/B test randomization

Step 2: Budget Partitioning

• Every participating ad campaign is split into virtual sub-campaigns
• Split is proportional to traffic split
  • 50/50 traffic split → 50/50 budget split
  • 70/30 traffic split → 70/30 budget split

Step 3: Isolated Auctioning

Treatment user visits site
        ↓
Only Treatment sub-campaign budget participates in auction

Control user visits site
        ↓
Only Control sub-campaign budget participates in auction

Why This Works

By isolating budgets, you create two independent mini-marketplaces:

Universe A (Treatment):    Universe B (Control):
- Treatment users          - Control users
- Treatment budget         - Control budget
- Treatment algorithm      - Control algorithm
        ↑                          ↑
   No interference between these two universes

Now if Treatment outperforms Control, you can confidently attribute it to the feature, not to budget cannibalization.

---

Step 4: Scaling Up — Orthogonal Concurrency

The New Problem

Budget Split solves one experiment. But what happens when you want to run two experiments simultaneously?

Experiment 1: Testing new bidding algorithm (variants: 1A, 1B)
Experiment 2: Testing new ad format (variants: 2A, 2B)

Running at the same time = budgets interfere across experiments

The Solution: Orthogonal Concurrency

Orthogonal means the experiments are mathematically independent — knowing a user's assignment in Experiment 1 tells you nothing about their assignment in Experiment 2.

For two concurrent experiments, each campaign budget is split into four buckets:

Experiment 1 →    1A          1A          1B          1B
Experiment 2 →    2A          2B          2A          2B
                ------      ------      ------      ------
Budget Bucket:   1A2A        1A2B        1B2A        1B2B

Visual representation:

                 Experiment 2
                  2A    2B
              ┌──────┬──────┐
Experiment  1A│ 1A2A │ 1A2B │
    1         ├──────┼──────┤
            1B│ 1B2A │ 1B2B │
              └──────┴──────┘

Each cell is an isolated budget bucket. Users land in exactly one cell.

Why "Orthogonal"?

When you analyze Experiment 1:

• Compare all 1A users (1A2A + 1A2B) vs all 1B users (1B2A + 1B2B)
• Experiment 2's variants are equally represented in both groups
• Therefore Experiment 2 cannot bias Experiment 1's results

The experiments are orthogonal — they do not distort each other.

---

Step 5: Residual Problems — What Still Goes Wrong

Even with Budget Split and Orthogonal Concurrency, perfect isolation is impossible. Here are the remaining interference sources you must understand:

Problem 1: Algorithmic Campaign Steering

Automated systems adjust bids/budgets based on COMBINED performance
        ↓
Treatment's performance affects Control's budget allocation
        ↓
Indirect interference remains

Problem 2: Pre-Experiment ML Contamination

ML models (pCTR, pCVR) train on historical data
        ↓
Historical data reflects pre-experiment distributions (mostly Control)
        ↓
Treatment group's model is trained on Control-influenced data
        ↓
Treatment starts with a biased model

Timeline of contamination:

Past 7-28 days          Experiment starts
[Control data] ────────→ [Treatment uses contaminated model]

Problem 3: Human Interventions

Ad operations staff manually adjust budgets based on aggregate metrics
        ↓
They cannot see the experiment split
        ↓
Their adjustments introduce uncontrolled interference

Problem 4: Right-Censoring

Campaign runs beyond experiment window
        ↓
Revenue from those campaigns is cut off mid-measurement
        ↓
Revenue metrics are artificially depressed or skewed

Fallback Strategy

When isolation cannot be guaranteed, the team pivots to:

• Causal inference methods (non-experimental counterfactual analysis)
• These estimate what would have happened without the treatment

---

Step 6: The Real-World Impact

Understanding the business value completes the picture:

Why This Matters Beyond Statistics

Reliable experiments
        ↓
Trustworthy results
        ↓
Faster decisions (fail fast on bad ideas)
        ↓
More innovation cycles per year
        ↓
Competitive advantage

---

Summary: The Complete Mental Model

PROBLEM:
Two-sided marketplace → shared budgets → SUTVA violation → Cannibalization Bias

SOLUTION LAYER 1 — Budget Split:
Split campaign budgets proportionally → isolated sub-campaigns per variant
→ Eliminates direct budget cannibalization

SOLUTION LAYER 2 — Orthogonal Concurrency:
Split into N×M budget buckets for N concurrent experiments
→ Experiments cannot interfere with each other

REMAINING CHALLENGES:
- Algorithmic steering
- ML training contamination  
- Human interventions
- Right-censoring

FALLBACK:
Causal inference when isolation fails

---

Quick Self-Check Questions

1. What does SUTVA stand for and what is its core requirement?
2. Explain in your own words how Cannibalization Bias occurs
3. If a campaign has a $900 budget and traffic is split 60/40, what are the two sub-budget amounts after Budget Split?
4. For three concurrent experiments (each with A/B variants), how many orthogonal budget buckets would you need?
5. Why does pre-experiment ML training data create interference even after Budget Split is implemented?

Answer to Q4: 2³ = 8 buckets (1A2A3A, 1A2A3B, 1A2B3A, 1A2B3B, 1B2A3A, 1B2A3B, 1B2B3A, 1B2B3B)