Step-by-Step Teaching

Step 1: Understand the Problem — Why Migrate at All?

Before learning how to migrate, understand why it becomes necessary.

The Core Tension

Systems are often built with tools that are fast and convenient early on, but those same tools can become liabilities as requirements evolve.

In this case:

• Redis was chosen because it was fast and familiar
• Over time, critical data (billing mappings) ended up stored there
• Redis was being used as an ephemeral cache — meaning data could disappear
• Losing a billing mapping meant a build would never be billed, with no way to recover it

The Key Lesson Here

The importance of data can outgrow the durability of the store holding it.

Ask yourself when designing systems:

• Is this data recoverable if lost?
• Is the storage tier appropriate for the data's criticality?

---

Step 2: Understand Why You Cannot Simply "Stop and Copy"

A naive migration looks like this:

1. Stop the system
2. Copy all data
3. Switch to new database
4. Restart

This works for systems that can tolerate downtime. Most production systems cannot.

In this case:

• Containers are constantly starting, polling, and expiring — 24/7
• There is no natural pause point
• Stopping would mean builds fail for real users

The Key Lesson Here

Live systems require live migrations — the migration must happen while the system keeps running.

This constraint forces you into a more careful, phased approach.

---

Step 3: Design the New Schema Around Access Patterns, Not the Old Schema

This is one of the most important conceptual steps.

The Wrong Approach

Map your old data structures directly to the new database.

The Right Approach

Start from what the code actually needs to do, then design the schema to serve those needs.

The team listed every operation the system performed:

1. Verify a token when a container polls
2. Add a token when a container starts
3. Expire tokens past their deadline
4. Count tokens to size the pool
5. Move a container between lifecycle statuses
6. Count containers by status
7. Look up the deployment behind a container (billing)
8. Remove expired containers

What This Analysis Revealed

• Almost every operation already knows the container ID
• Only token verification starts from a token, not a container
• Therefore: center the schema on the container

Old Redis model (data structures implied behavior):
- Tokens → set + sorted set
- Statuses → three separate sorted sets
- Billing mapping → string

New DynamoDB model (explicit keys and indexes):
type ContainerRecord = {
  warmPoolId: string   // partition key
  containerId: string  // sort key
  token: string        // stored as a hash (never exposed raw)
  status: 'pending' | 'polling' | 'building'
  expiresAt: number    // single expiry field
}

What Changed and Why

The Key Lesson Here

Schema design should follow access patterns, not the shape of the old system.

When migrating, treat it as an opportunity to redesign, not just translate.

---

Step 4: Execute a Phased Rollout with Rollback at Every Step

This is the operational heart of a safe live migration.

The Five Phases

Phase 1: Redis Only
└── Baseline established. Normal behavior recorded.

Phase 2: Dual Writes
└── Write to BOTH Redis and DynamoDB
└── Redis = source of truth
└── DynamoDB failures = logged, not fatal
└── Rollback: stop writing to DynamoDB

Phase 3: Shadow Reads
└── Read from BOTH stores
└── Compare results
└── Mismatches = investigated, not ignored
└── Rollback: stop reading from DynamoDB

Phase 4: DynamoDB Primary
└── DynamoDB = source of truth
└── Redis writes kept as backup
└── Rollback: flip reads back to Redis

Phase 5: DynamoDB Only
└── Redis writes removed
└── No easy rollback (but tokens expire in ~10 min anyway)

Why This Structure Works

Each phase answers a specific question before you commit further:

The Key Lesson Here

Each phase should have a rollback plan and a clear signal (dashboards, metrics) that tells you when it is safe to advance.

Never advance based on assumption. Advance based on evidence.

---

Step 5: Validate with Shadow Mode — Tests Are Not Enough

Shadow reads are a powerful technique worth understanding deeply.

What Shadow Mode Does

• Every read goes to both databases simultaneously
• Results are compared automatically
• Mismatches are logged and investigated

What Tests Cannot Tell You

Tests verify behavior in controlled conditions. Shadow mode verifies behavior under real production traffic, including:

• Timing edge cases
• Concurrent writes
• Unexpected data states
• Race conditions in dual-write scenarios

What to Watch on Dashboards

The team monitored:

• Match rates — do both stores return the same values?
• Write errors — are DynamoDB writes failing silently?
• Per-query latency — how does DynamoDB compare to Redis?
• Expiration counts — is stale data being cleaned up correctly?

The Key Lesson Here

Shadow mode surfaces bugs that tests cannot, because production traffic is more complex than any test suite.

Every mismatch is a bug to fix before advancing, not a number to explain away.

---

Step 6: Identify Hidden Assumptions — The Hardest Part

This is where the migration became genuinely difficult, and where the deepest lesson lives.

The Hidden Assumption

The supply loop (which refills the warm pool) made hundreds of database reads per run. Nobody wrote this down as a requirement. It just worked because Redis was fast enough.

Loop as designed (implicit assumption: reads are ~1ms):
check → create → check → create → check → create → ...
(one read before every single container)

When DynamoDB replaced Redis:

• Redis P95 latency for that check: 1.29ms
• DynamoDB P95 latency for that check: 5.13ms
• Multiplied across hundreds of calls per loop run → loop stalled for minutes

Why This Is Dangerous

The assumption was never documented. It was embedded in the design of the loop itself. The system worked, so no one questioned it.

This is called an implicit performance dependency — the code depends on a specific performance characteristic of a component, but that dependency is never stated.

How to Find Hidden Assumptions

Ask these questions during any migration:

1. What latency does each component assume from its dependencies?
2. What call volume is being made to each dependency?
3. Are there any N+1 patterns — operations that repeat per item rather than per batch?
4. What would break if any dependency became 2x slower? 10x slower?

The Key Lesson Here

The hardest part of migration is not moving the data. It is finding the assumptions built on top of the old system.

---

Step 7: Redesign Around the New Reality, Not the Old One

Once the hidden assumption was found, there were two options:

Option A: Compensate (Batching)

Check state once per N containers, where N is derived from the latency ratio.

Batch design:
check → create, create, create, ... (17 at a time) → check → ...

Why they rejected it:

• The constant 17 was derived from a measured latency ratio
• That ratio changes with load
• This would create another hidden assumption — just a different one

Option B: Redesign (Concurrency)

Remove the serial dependency entirely. Let pool supply calls run concurrently.

Concurrent design:
Pool A: check → create
Pool B: check → create    ← all running at the same time
Pool C: check → create

Why this is better:

• No constant tied to latency
• No serialization on any single read
• Accepted tradeoff: occasionally creates a few extra containers from a slightly stale view — documented and accepted

The Key Lesson Here

When a dependency changes, redesign to remove the dependency rather than compensate for it.

Compensation preserves the fragility. Redesign eliminates it.

---

Step 8: Synthesize — The Complete Mental Model

Here is the full framework this migration teaches:

LIVE DATABASE MIGRATION FRAMEWORK

1. JUSTIFY
   └── Is the current store appropriate for the data's criticality?

2. DESIGN
   └── List all access patterns first
   └── Design schema to serve those patterns, not to mirror old schema

3. PHASE
   └── Dual writes → Shadow reads → New primary → Old removed
   └── Rollback available at every phase except the last

4. VALIDATE
   └── Shadow mode catches what tests miss
   └── Dashboards decide when to advance, not schedules

5. HUNT ASSUMPTIONS
   └── What latency does the code silently depend on?
   └── Where are the N+1 patterns?
   └── What would 5x slower reads break?

6. REDESIGN
   └── Remove dependencies rather than compensate for them
   └── Document accepted tradeoffs explicitly

---

Final Summary

The team's own summary said it best:

"Even a 1ms-to-15ms query time degradation on P90 could bring down our warm pool management logic."

Nobody knew that sentence was true until the migration forced them to find it.