Step-by-Step Study Material

Step 1: Understanding the Problem Space

What is Batch Compute?

Batch compute refers to workloads that:

• Run to completion (unlike long-running services)
• Are queued and executed based on priority and available capacity
• Need resource management across multiple teams/applications

Example batch workloads:
├── Data processing jobs
├── ML training runs
├── Report generation
└── ETL pipelines

Netflix's Original Solution: CMB (Compute Managed Batch)

Built in 2018, CMB provided:

---

Step 2: Core Concepts — Tenants and Capacity

Tenant Types

Think of tenants like a company org chart:

Organization (Internal Tenant)
├── Team A (Internal Tenant)
│   ├── App 1 (Leaf Tenant) ← actually runs jobs
│   └── App 2 (Leaf Tenant)
└── Team B (Internal Tenant)
    └── App 3 (Leaf Tenant)

Key Rule: Only leaf tenants submit actual jobs. Internal tenants organize and share capacity downward.

Two Types of Capacity

Reserved Capacity

┌─────────────────────────────────────┐
│         Reserved Capacity           │
│                                     │
│  Internal Tenant → shared across    │
│  all subtree leaf tenants           │
│                                     │
│  Leaf Tenant → exclusively yours,   │
│  NOT shared with others             │
└─────────────────────────────────────┘

Shared Capacity

┌─────────────────────────────────────┐
│    Global Shared Capacity Pool      │
│                                     │
│  • Any tenant can "burst" into it   │
│  • No reservation required          │
│  • Fair-shared at admission time    │
│  • ⚠️ No preemption in old CMB      │
└─────────────────────────────────────┘

Critical Limitation of CMB: Once a job was admitted, it ran to completion — even if fair-share demand shifted dramatically. This caused poor resource utilization.

---

Step 3: Why Migrate? — Identifying the Pain Points

CMB's Growing Problems

Problem 1: Built before modern Kubernetes ecosystem matured
    → Open-source tools now offer what CMB built custom

Problem 2: Feature development was difficult
    → CMB was too far removed from underlying Kubernetes clusters
    → Adding preemption was especially cumbersome

Problem 3: No preemption
    → Resources couldn't be reclaimed once allocated
    → Lower-priority jobs could block higher-priority ones

What the Kubernetes Ecosystem Now Offers

Modern tools provide out-of-the-box:

• ✅ Fair sharing
• ✅ Hierarchical tenants
• ✅ Capacity management
• ✅ Priority queuing
• ✅ Preemption

---

Step 4: The Solution — Introducing Kueue

What is Kueue?

Kueue is a cloud-native job queueing system for Kubernetes batch workloads.

Key Kueue Concepts and How They Map to CMB

Visual Mapping

CMB Structure          →      Kueue Structure
─────────────────────────────────────────────
Internal Tenant        →      Cohort
  └── Leaf Tenant      →      ClusterQueue
        └── Jobs        →      LocalQueue → Workloads

---

Step 5: The Migration Architecture

Old Flow vs New Flow

OLD FLOW (CMB):
User → CMB Queue → CMB Scheduler → Titus Cell

NEW FLOW (Netflix Batch / Kueue):
User → Titus Endpoint → Kueue Router → Kueue-enabled Cell
                              ↑
                         (Kueue handles
                          queuing & scheduling)

Key Architectural Insight

Netflix uses Titus federation — a single endpoint that routes jobs across multiple Kubernetes clusters (cells). The Kueue router directs jobs to the appropriate Kueue-enabled cell.

Migration Process (Operator Perspective)

The migration was designed to be:

• Simple: Toggle a button in the UI per tenant
• Reversible: Easy rollback if issues arise
• Automatic under the hood:

Click "Enable Kueue" on tenant
         ↓
Internal Tenants  → converted to Cohorts
Leaf Tenants      → converted to ClusterQueue + LocalQueue
Capacity Config   → converted to Resource Flavors + Nominal Quotas

---

Step 6: The Game Changer — Fair Sharing and Preemption

What Changed with Kueue

Preemption-Based Fair Sharing

Scenario: Team A has reserved capacity but isn't using it
                    ↓
Kueue allows Team B to borrow that idle capacity
                    ↓
When Team A needs it back → Team B's jobs are PREEMPTED
                    ↓
Team A reclaims its reserved resources

This is fundamentally different from CMB where borrowed resources could never be reclaimed.

Reading the Kueue Configuration

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "team-a-cq"        # This team's ClusterQueue
spec:
  preemption:
    reclaimWithinCohort: Any          # Can reclaim from ANY tenant in cohort
    withinClusterQueue: LowerPriority # Can preempt lower-priority jobs within own queue

Breaking down the preemption rules:

Business Benefits

For Tenants:
├── Use more idle capacity from other reservations
├── Submit more jobs without risk of starvation
└── Faster turnaround for business-critical (high-priority) workloads

For Netflix Platform:
└── Significant increase in average resource utilization

---

Step 7: Synthesis — The Big Picture

Why This Migration Matters

Before (CMB)                    After (Kueue)
─────────────────────────────────────────────────
Custom-built scheduling    →    Kubernetes-native
No preemption              →    Priority preemption
Static resource allocation →    Dynamic fair sharing
Hard to extend             →    Ecosystem-backed
Poor utilization           →    Significantly higher utilization

Broader Impact

• Learnings shared with ML training infrastructure teams
• Foundation for more Kubernetes-native workloads at Netflix
• Millions of batch jobs now managed by Kueue in production

---

Quick Knowledge Check

Test yourself with these questions:

1. What is the difference between a leaf tenant and an internal tenant?

   Leaf tenants submit actual jobs; internal tenants organize hierarchy and share capacity downward

2. Why was preemption impossible in CMB?

   Once admitted, jobs ran to completion regardless of changing fair-share demand

3. What does reclaimWithinCohort: Any mean in practice?

   A tenant can reclaim its reserved capacity from any other tenant currently borrowing it

4. What Kueue object replaces a CMB internal tenant?

   A Cohort

5. What is the key utilization benefit of preemption-based fair sharing?

   Idle reserved capacity can be lent out and reclaimed, preventing waste while maintaining guarantees