How Netflix Automatically Right-Sizes Media Containers

Defined Learning Outcomes

After studying this material, you should be able to:

1. Explain why manual resource configuration fails at scale in serverless platforms
2. Describe the end-to-end architecture of an automated right-sizing system
3. Identify the five stages of a recommendation lifecycle
4. Understand how safety mechanisms prevent failures during automated changes
5. Apply the lessons learned to design similar systems

---

Step 1: Understanding the Problem

What is Over-Provisioning?

When engineers deploy containers, they must specify resource limits:

numCpus: 8
memoryInMB: 18000
diskSizeInMB: 50000
networkInMbps: 1000

The natural human tendency is to guess high. Engineers either:

• Copy settings from existing functions
• Size for the absolute worst-case scenario
• Never revisit settings after initial deployment

Why Manual Tuning Fails at Scale

Think of it like setting a speed limit on every road in a country — manually, one by one:

Key Insight: Over-provisioning is not just wasteful — it compounds over time as new workloads are added and old configs are never cleaned up.

---

Step 2: The Solution Architecture Overview

What is the Stratum Resource Tuner (SRT)?

SRT is an automated pipeline that:

1. Observes real resource usage
2. Generates recommendations
3. Tests them safely
4. Applies them automatically
5. Monitors and rolls back if something goes wrong

The Five Stages of a Recommendation

CREATED → EVALUATING → APPLIED
                    ↘ REJECTED

Think of this like a drug approval process:

• Lab research (data collection)
• Clinical trials (synthetic + production experiments)
• Approval (apply to production)
• Post-market surveillance (continuous monitoring)
• Recall if needed (automated rollback)

---

Step 3: Data Collection — Knowing What's Actually Used

Two Data Stores Serve Different Purposes

Elasticsearch stores the high-level summary per execution:

{
  "tags.functionName": "VideoEncoder.encodeV1",
  "tags.cpuCount": 8,
  "tags.memoryInMB": 18000,
  "tags.stratum.success": true
}

Iceberg stores the detailed time-series data:

{
  "cpu_requested": 8,
  "resource_usage": [
    { "cpu_usage": 1.13, "memory_mb": 3701 },
    { "cpu_usage": 2.25, "memory_mb": 3584 },
    { "cpu_usage": 4.16, "memory_mb": 6750 }
  ]
}

Why Two Stores?

Key Insight: You cannot right-size what you cannot measure. Sub-minute resolution data is essential for catching peak usage spikes.

---

Step 4: Generating Recommendations

The Daily Workflow Logic

Every day, a scheduled job reads 30 days of historical data and applies this logic:

For each function + resource type:
  1. Calculate p99 and p99.9 usage percentiles
  2. Add a safety buffer
  3. Compare to current allocation
  4. If savings exist → create recommendation

Real Example: VideoEncoder.encodeV1

Memory recommendation generated:

{
  "current_value": 18000,
  "recommended_value": 15300,
  "stats": {
    "usage": {
      "avg": 5296,
      "p99": 6750,
      "p999": 7820
    }
  }
}

Reading this data:

• Average usage: ~5,296 MB (only 29% of 18,000 MB allocated)
• Even at p99.9: 7,820 MB — still far below 18,000 MB
• Recommended: 15,300 MB (a conservative reduction with buffer)

Key Insight: Using percentiles (p99, p99.9) rather than averages protects against rare spikes while still enabling significant reductions.

---

Step 5: The Safety Net — Retry With More Resources

The Problem Recommendations Cannot Fully Solve

Historical data cannot predict future unusual inputs. A video encoder might occasionally receive an unusually complex file that needs more memory than typical.

The Watcher Mechanism

Each container runs a watcher process that monitors usage in real time:

If memory_usage > 85% of limit:
    Cancel execution
    Retry on container with: new_size = original_size × 1.5

If disk_usage > 95% of limit:
    Cancel execution
    Retry on container with: new_size = original_size × 1.5

Visualizing this for VideoEncoder.encodeV1:

Normal run:     [=====>          ] 15,300 MB limit — completes fine
Heavy input:    [===============>] Hits 85% threshold
                ↓
Retry on:       [=====>               ] 22,950 MB container

Key Insight: This decouples "typical case optimization" from "edge case handling." You optimize for the 99.9% of runs while still safely handling outliers.

---

Step 6: Two-Stage Verification Before Any Change Goes Live

Why Two Stages?

A single test is not enough because:

• Synthetic tests are controlled but artificial
• Production tests are real but riskier

Both must pass before a recommendation is applied.

Stage 1: Synthetic Experiment

Run 3,000 executions:
  - Baseline cluster: current config (18,000 MB)
  - Experimental cluster: new config (15,300 MB)

Check:
  ✓ Slowness increase < threshold?
  ✓ Retry rate < threshold?

Stage 2: Production Experiment

Split real traffic:
  - Baseline: existing containers
  - Experimental: right-sized containers
  - Minimum 500 samples each

Check same two conditions.

VideoEncoder.encodeV1 Results

"syntheticExperiment": {
  "result": "SUCCESS",
  "slownessIncreasePct": 0.0,
  "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
},
"productionExperiment": {
  "result": "SUCCESS",
  "slownessIncreasePct": 1.0,
  "retryRatePct": { "baseline": 0.0, "experimental": 0.0 }
}

1% slowness increase is acceptable. Zero retry rate increase confirms safety.

---

Step 7: Post-Apply Monitoring and Automated Rollback

The Problem That Emerged

After applying a recommendation to VideoEncoder.encodeV2, an unexpected spike occurred. Because early versions had no automated rollback, engineers had to manually intervene — slow and risky.

The Solution: Two Monitoring Windows

Fast window (30 minutes):
  Alert threshold: retry rate > 3%
  Purpose: Catch sudden spikes quickly

Slow window (24 hours):
  Alert threshold: retry rate > 2%
  Purpose: Catch gradual degradation

Normal baseline: ~0.1% retry rate Acceptable variance: up to 2–3% Auto-revert trigger: exceeds thresholds above

The Rollback Logic

If threshold breached:
    Immediately restore previous configuration
    (Do NOT attempt dynamic adjustment — keep it simple)

Key Insight: Simple, predictable rollback is safer than clever dynamic adjustment during an incident. Speed of recovery matters more than elegance.

---

Step 8: Lessons Learned — What Went Wrong and How It Was Fixed

The VideoEncoder.encodeV2 Incident

What happened: A large single-step reduction was applied. An unexpected workload pattern caused retry rates to spike. Manual recovery was slow.

Three changes made as a result:

The Incremental Step Approach Visualized

Old approach:
18,000 MB ──────────────────→ 15,300 MB (one jump)

New approach:
18,000 MB → 15,300 MB → 13,005 MB → 11,054 MB
           (confirm)    (confirm)    (confirm)
           1 week wait  1 week wait  1 week wait

Key Insight: Incremental changes with mandatory cooldown periods allow you to observe real production behavior before taking the next step.

---

Step 9: Results and Impact

VideoEncoder.encodeV1 Outcomes

CPU:    8 cores → 4 cores    (50% reduction)
Memory: 18,000 MB → 15,300 MB (40% reduction — first step)

Performance impact: None measurable
Retry rate impact:  None

System-Wide Goal

• Initial target: 5% reduction in overall Stratum compute costs
• Achieved through automating what was previously impossible to do manually at scale

---

Summary: The Complete Mental Model

┌─────────────────────────────────────────────────────┐
│              SRT Pipeline Overview                  │
├─────────────────────────────────────────────────────┤
│  MEASURE    → Collect sub-minute usage data         │
│  ANALYZE    → Calculate p99/p99.9 + safety buffer   │
│  RECOMMEND  → Generate CREATED recommendation       │
│  TEST       → Synthetic → Production experiments    │
│  APPLY      → Update production configuration       │
│  MONITOR    → Watch retry rates for 90 days         │
│  REVERT     → Auto-rollback if thresholds breached  │
└─────────────────────────────────────────────────────┘

The Three Core Design Principles

1. Measure before acting — never guess, always use real data
2. Test before committing — two-stage verification catches most problems
3. Monitor after applying — production behavior can always surprise you

---

Self-Check Questions

1. Why is p99.9 used instead of average usage for recommendations?
2. What triggers the "retry with more resources" mechanism, and why 1.5x?
3. Why does SRT use two separate monitoring windows (30-minute and 24-hour)?
4. What was the key lesson from the VideoEncoder.encodeV2 incident?
5. Why is rollback logic kept simple rather than dynamic?