After studying this material, you should be able to:

1. Explain what Netflix Conductor is and what problems it solves
2. Describe how Conductor's architecture evolved across versions
3. Understand the key engineering decisions behind scalability improvements
4. Identify the developer experience improvements introduced by the Workflow SDK
5. Recognize execution control mechanisms like Run Strategies and managed tasks

---

Step-by-Step Study Material

Step 1: What Is Conductor and Why Does It Exist?

The Core Problem

Modern applications at scale are built from microservices: small, independent services that each do one thing. But business processes require many steps in sequence or parallel.

Example: Publishing a Netflix title requires:

• Ingesting camera files
• Validating content
• Transcoding video
• Delivering to global servers

Without orchestration, coordinating these steps means:

• Manual retry logic
• No visibility into failures
• Fragile custom coordination code

The Solution: Workflow Orchestration

Conductor acts as a central coordinator that:

[Service A] → [Service B] → [Service C]
     ↑               ↑              ↑
     └───────[Conductor]────────────┘
              (tracks state,
               retries failures,
               provides visibility)

Key Definition: A workflow is a declared graph of tasks. A task is one unit of work performed by a microservice.

Scale Context

---

Step 2: The Original Architecture and Its Limits

First Generation Stack

┌─────────────────────────────────┐
│         Conductor Engine        │
├─────────────────────────────────┤
│  DynoQueues  │  Dynomite (Redis)│  ← Queueing + Storage
│  Elasticsearch                  │  ← Indexing
│  Cassandra (partial)            │  ← Some execution data
└─────────────────────────────────┘

Why It Failed at Scale

Problem 1: Dynomite is in-memory

• Built on Redis → designed for caching, not durable storage
• Scaling requires adding more memory and nodes
• Each replica holds a full copy of all data → expensive

Problem 2: DynoQueues had uneven scheduling

• Zone-sharded design caused some zones to get more work than others

Problem 3: Synchronous evaluation with race conditions

• Workflow state was mutated in multiple places simultaneously
• Competing updates could leave workflows in inconsistent states
• No automated repair mechanism

---

Step 3: The 2.x Evolution — Fixing the Foundation

Replacement 1: Cassandra as Primary Datastore

Before:  Dynomite (in-memory, limited scale)
After:   Cassandra (disk-based, horizontally scalable)

Why Cassandra?

• Scales horizontally by adding nodes (not just memory)
• No zone-affinity constraints
• Durable storage for millions of workflows

Additional change: Large task inputs/outputs moved to S3, keeping bulky data off the primary database.

Replacement 2: Timestone Queue

Before:  DynoQueues (built on Dynomite, uneven)
After:   Timestone (high-throughput, low-latency)

Timestone added two critical capabilities:

1. Delayed queuing — schedule work for the future
2. Background repair — automatically detect and recover stuck workflows

Replacement 3: Decoupled Indexing

Before:  Indexing on the critical path (slows every operation)
After:   Kafka buffer → independent indexer → Elasticsearch

This means indexing failures no longer block workflow execution.

Conductor 3.0 Summary

┌─────────────────────────────────────┐
│         Conductor 3.0 Stack         │
├─────────────────────────────────────┤
│  Timestone          │  Queue        │
│  Cassandra          │  Execution    │
│  Kafka → ES         │  Indexing     │
│  Iceberg            │  Long-term    │
│  Multi-cluster      │  Isolation    │
└─────────────────────────────────────┘

---

Step 4: The Evaluation Engine Rewrite

This is the most technically deep section. Take it in parts.

Problem A: Memory Bloat

Old approach:

Evaluate workflow → Load ENTIRE workflow state into memory
                  → Decide next step
                  → Write back

For a workflow with 10,000 tasks, this means loading all 10,000 tasks just to advance one step.

New approach:

Evaluate workflow → Load lightweight "blueprint"
                  → Load ONLY the specific task data needed
                  → Decide next step

Result: 40% reduction in p99 evaluation latency, and support for 30,000 tasks (up from 2,500).

Problem B: Race Conditions on Task State

The scenario:

Worker sends:  IN_PROGRESS → IN_PROGRESS → COMPLETED
                                              ↑
Network reorders: late IN_PROGRESS arrives AFTER COMPLETED
                                              ↓
Result: Task reverts to IN_PROGRESS ← BUG!

First attempt: Cassandra Lightweight Transactions (LWT)

• LWT serializes writes (only one wins)
• Problem: LWT and regular writes use different timestamping mechanisms
• Mixing them breaks consistency → anti-pattern

Final solution: Two-partition model

┌─────────────────┐    ┌──────────────────┐
│ Pending Partition│    │Terminal Partition │
│ (in-flight tasks)│    │(completed/failed) │
└─────────────────┘    └──────────────────┘

Reconciliation rule (simple and elegant):

• Terminal state always wins over non-terminal
• Between two writes of the same type → last-write-wins

Race scenario with new model:
COMPLETED → written to Terminal Partition ✓
Late IN_PROGRESS → written to Pending Partition
On read: Terminal wins → task stays COMPLETED ✓

Problem C: Synchronous Evaluation and Lock Contention

Old flow:

Task completes → Request evaluates workflow inline
              → Must acquire lock on workflow
              → Other updates WAIT for lock
              → Under load: lock contention → minutes of delay

New flow (Async Evaluation):

Task completes → Enqueue workflow onto Timestone exclusive queue
              → Central async evaluator processes in order
              → Only ONE worker per workflow at a time
              → Lock contention: essentially zero

Key insight: The queue is the lock. Because it's exclusive, ordering is guaranteed without explicit locking.

---

Step 5: Execution Control — Run Strategies

The Problem Without Run Strategies

Imagine two workflow instances both processing the same movie simultaneously:

Instance A: reads movie data → modifies → writes
Instance B: reads movie data → modifies → writes  ← overwrites A!

Teams had to build custom locking logic themselves — fragile and complex.

The Solution: Declare Concurrency Policy on the Workflow

@RunStrategy(type = SEQUENTIAL, sequenceKey = "movieId")
@WorkflowMethod(name = "ingest_movie")
default void ingestMovie(MovieInput input) { ... }

Two strategies:

The sequence key is critical:

Movie A (id=101): Run 1 → Run 2 → Run 3  (serial)
Movie B (id=202): Run 1 → Run 2 → Run 3  (serial)
         ↕
Movies A and B run CONCURRENTLY with each other

---

Step 6: Managed Tasks — Shifting Compute Responsibility

Traditional Model: Polled Workers

Your Application                    Conductor
┌──────────────┐                   ┌──────────┐
│ Worker Fleet │ ←── polls ──────→ │  Server  │
│ (you operate)│ ──── result ────→ │          │
└──────────────┘                   └──────────┘

You must: deploy, scale, and maintain the worker fleet.

New Model: Managed Tasks

Conductor drives the compute directly:

TITUS Task (containers):

Workflow step → Conductor launches Docker container on Titus
             → Monitors job
             → exit code 0    → COMPLETED
             → exit code ≠ 0  → FAILED
             → timeout        → FAILED

STRATUM Task (serverless media functions):

Workflow step → Conductor invokes Stratum function
             → Waits for result
             → COMPLETED / FAILED / TIMED_OUT

Mixing all three in one workflow:

[Polled Worker]     [Titus Container]    [Stratum Function]
(steady volume)  →  (heavy batch step) → (media processing)

---

Step 7: Dynamic Thread Allocation

The Static Thread Problem

Task Queue A: ████████████████ (1000 items backlogged)
Task Queue B: ░░░░░░░░░░░░░░░░ (empty)

Thread allocation (fixed):
Queue A: 10 threads  ← not enough!
Queue B: 10 threads  ← wasted!

Result: CPU flat, queues backing up

The Dynamic Solution

Each polling interval:

Step 1: Poll Timestone for current depth of ALL queues
Step 2: Split thread pool into two parts:

┌─────────────────────────────────────────┐
│ Minimum Allocation: every queue gets    │
│ a baseline (prevents starvation)        │
├─────────────────────────────────────────┤
│ Floating Pool: remaining threads        │
│ distributed proportional to queue depth │
└─────────────────────────────────────────┘

Step 3: Apply multiplier ceiling
(max growth per cycle = configurable factor)
→ prevents sudden flood of downstream services

Result:

• Threads go where work actually is
• CPU becomes an accurate signal → autoscaling works correctly
• No queue is ever fully starved

---

Step 8: The Workflow SDK — Type-Safe Authoring

The Problem with JSON Workflows

Type safety issue:

// JSON declares: "movieId": "${task.output.movieId}"
// Worker reads:
long movieId = (Integer) inputData.get("movieId"); // ClassCastException at runtime!
long movieId = (Integer) inputData.get("movieid"); // NullPointerException! (typo)

Verbosity issue:

• A two-task workflow = dozens of lines of JSON
• Real workflows = hundreds of lines, hard to review

The SDK Solution

Two core building blocks:

How it works:

You write Java → Gradle plugin → JSON definition → Conductor server
                 (build time)    (same as before)

The server never sees Java. It gets the same JSON it always understood.

Type-safe example:

// Before (JSON + unsafe worker):
long movieId = (Integer) inputData.get("movieId"); // fingers crossed

// After (SDK):
@TaskMethod
public Movie getMovie(long movieId) { /* compiler checks types */ }

// Workflow wiring:
Movie movie = getMovie(input.getMovieId()); // compiler verified ✓
return wrapMovie(movie);                    // compiler verified ✓

Expressing Complex Patterns

Loops (always do-while in Conductor):

do {
    status = checkStatus(jobId);
} while (status.getState() == "PENDING");

Parallel execution:

Fork<Movie> details = ParallelExecution.fork(() -> getMovie(id));
Fork<Artwork> art   = ParallelExecution.fork(() -> getArtwork(id));
ParallelExecution.allOf(details, art); // FORK + JOIN

Sub-workflows: Just call another @WorkflowMethod like a regular method.

---

Step 9: Putting It All Together

Here is how all the pieces connect:

┌─────────────────────────────────────────────────────┐
│                  DEVELOPER EXPERIENCE                │
│  Workflow SDK (type-safe Java DSL)                   │
│  IDE plugins, testing framework                      │
└──────────────────────┬──────────────────────────────┘
                       │ generates JSON
┌──────────────────────▼──────────────────────────────┐
│                  EXECUTION CONTROL                   │
│  Run Strategies (Sequential / Parallel)              │
│  Managed Tasks (Titus / Stratum / Polled Workers)    │
└──────────────────────┬──────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────┐
│                  EVALUATION ENGINE                   │
│  Async evaluation (no lock contention)               │
│  Blueprint-based (low memory footprint)              │
│  Two-partition task state (no race conditions)       │
└──────────────────────┬──────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────┐
│                    DATA PLANE                        │
│  Timestone (queuing + async eval queue)              │
│  Cassandra (execution data)                          │
│  Kafka → Elasticsearch (indexing)                    │
│  S3 (large payloads) + Iceberg (long-term)           │
└─────────────────────────────────────────────────────┘

---

Quick Reference: Key Concepts