Step-by-Step Teaching

Step 1: Foundation — What Is a Sandbox?

Before understanding the findings, you need the mental model.

┌─────────────────────────────────┐
│           HOST MACHINE          │
│  (physical server, cloud infra) │
│                                 │
│  ┌──────────┐  ┌──────────┐    │
│  │ microVM  │  │ microVM  │    │
│  │ Sandbox A│  │ Sandbox B│    │
│  │(Customer)│  │(Customer)│    │
│  └──────────┘  └──────────┘    │
└─────────────────────────────────┘

Key concept: Multiple customers share one physical host. The sandbox is supposed to create an isolation boundary — what happens inside should stay inside.

Vercel uses Firecracker microVMs — lightweight virtual machines designed specifically for security isolation.

---

Step 2: The Most Critical Concept — Trust Boundaries

This is the single most important idea in the entire article.

Definition

A trust boundary is the line between code/data you control and code/data you do NOT control.

The Golden Rule

Host should NEVER trust input from the Guest
because the Guest runs UNTRUSTED customer code

Why This Matters — Real Example from the Article

FLAWED DESIGN:
┌─────────────────────────────────────────┐
│ Guest (sandbox) takes a snapshot        │
│ Guest reports back: {url, size, checksum}│
│ Host TRUSTS this response ✗             │
└─────────────────────────────────────────┘

FIXED DESIGN:
┌─────────────────────────────────────────┐
│ Guest (sandbox) takes a snapshot        │
│ Host generates its OWN storage path     │
│ Host validates everything server-side ✓ │
└─────────────────────────────────────────┘

Think of it like this: Imagine a bank letting customers write their own account balances on a form and trusting those numbers. That is exactly what trusting guest-reported values means.

---

Step 3: Understanding the Vulnerabilities Found

Vulnerability 1 — Linux Kernel Zero Day (Critical)

What is a zero day?

A vulnerability that is unknown to the vendor and has no patch available — zero days to fix it.

What happened:

Attacker's microVM
        │
        │ sends specially crafted packets
        ▼
Linux Kernel Networking Stack
        │
        ├──► Leak host memory into outgoing packets
        ├──► Crash the entire host machine
        └──► Execute code with FULL admin privileges on host

Why is this especially dangerous?

Almost every major cloud provider (AWS, Google, Azure, etc.) uses this same Linux kernel layer. One bug = shared risk across the entire industry.

Severity scale:

---

Vulnerability 2 — Cross-Tenant Disk Block Leak

Found by: A Vercel intern (important lesson — threats come from anywhere)

The flaw:

Sandbox A gets deleted
    │
    │ disk blocks get "recycled"
    ▼
Sandbox B gets created
    │
    │ can READ old disk blocks from Sandbox A
    ▼
Sandbox B reads Sandbox A's leftover data ✗

Real-world analogy: Like renting a hotel room and finding the previous guest's documents still in the safe because staff forgot to clear it.

---

Vulnerability 3 — Cross-Tenant Snapshot Swap (High)

This is the most instructive example because it shows how vulnerabilities chain together.

Step-by-step attack breakdown:

Step 1: Researcher owns Account A and Account B

Step 2: Account A takes a snapshot
        Guest reports back: {storage_url_A, size, checksum}
        Host TRUSTS this ← THE FLAW

Step 3: Researcher intercepts the response inside their sandbox
        Rewrites it: {storage_url_B, size, checksum}
        (swapping Account B's storage location)

Step 4: Restoring "Account A's snapshot" actually
        loads Account B's entire filesystem

Result: Full read/write access to another account's files

Why the attack had limits:

To target a STRANGER you would need their snapshot storage key
= random 96-bit identifier
= never exposed through the API
= only exists inside the victim's own sandbox

∴ No demonstrated path to target arbitrary victims

But the fix was still urgent because:

A security boundary that depends on a secret staying secret is not a real boundary. Secrets can leak. Design must assume they will.

---

Step 4: The Three Structural Security Principles

Trail of Bits gave three recommendations. These are universal security design principles you should memorize.

Principle 1 — Eliminate All Guest Trust

BEFORE:  Host ←── trusts ──── Guest values
AFTER:   Host derives values server-side
         OR signs them with keys guest cannot reach

Practical rule: Every value crossing from guest → host must be treated as attacker-controlled input.

---

Principle 2 — One Policy, One Decider

The problem:

Network Policy
    │
    ├──► Subsystem A parses it → decision X
    ├──► Subsystem B parses it → decision Y  ← different answer!
    └──► Subsystem C parses it → decision Z

Why this is dangerous: Attackers find the subsystem that makes the most permissive decision and exploit the gap.

The fix:

Network Policy
    │
    └──► ONE canonicalization routine
              │
              └──► ONE decision component
                        │
                        └──► Consistent enforcement everywhere

Real-world analogy: Imagine a nightclub with three different bouncers using three different guest lists. An attacker finds the most lenient bouncer.

---

Principle 3 — Assume the Next Zero Day

This is a defense-in-depth mindset:

Do not assume your isolation is perfect. Build layers so that when (not if) a vulnerability is found, the damage is contained.

Vercel's implementation:

Layer 1: Firecracker microVM isolation
Layer 2: Host kernels kept up to date
Layer 3: Nested virtualization disabled (reduces attack surface)
Layer 4: Firecracker's jailer = restricted permissions per microVM

Mental model — Swiss cheese defense:

Each layer has holes, but holes rarely line up
┌──┐ ┌──┐ ┌──┐ ┌──┐
│  ○ │  │ ○  │  │  │  ← attacker needs ALL holes to align
└──┘ └──┘ └──┘ └──┘

---

Step 5: Responsible Disclosure Process

This is how security research should work ethically.

1. Researcher finds vulnerability
        │
        ▼
2. Researcher contacts vendor PRIVATELY
        │
        ▼
3. Vendor + Researcher collaborate on fix
        │
        ▼
4. Vendor notifies affected industry partners
   (AWS, other cloud providers in this case)
        │
        ▼
5. Patch is developed and reviewed
        │
        ▼
6. Patch ships publicly
        │
        ▼
7. Vulnerability details published
   (CVE assigned, public disclosure)

Why not disclose immediately?

Publishing details before a patch exists gives every attacker a roadmap with no defense available.

---

Step 6: Bug Bounty Programs — What the Numbers Tell Us

1,285 total reports
    │
    ├── 877 duplicates (68%) ← same bugs found repeatedly
    ├── 319 not applicable (25%) ← misunderstood scope
    ├──  57 valid (4.4%) ← real vulnerabilities
    └──  32 informative (2.5%) ← useful but not exploitable

Key insight: High duplicate rate means researchers converge on the same obvious attack surfaces. Unique findings come from diverse approaches.

Severity breakdown of valid findings:

$242,000 paid — mostly for that one Critical finding, demonstrating how severity drives bounty value.

---

Summary — Core Concepts to Remember

---

Self-Test Questions

1. Why can a host machine never trust values reported by a guest sandbox?
2. What made the snapshot swap attack limited in scope? Why was it still fixed urgently?
3. Explain the "one policy, one decider" principle in your own words
4. What is the difference between a zero day and a regular vulnerability?
5. Why does "assume the next zero day" lead to better security design than "prevent all zero days"?