After studying this material, you should be able to:

1. Explain why tiered storage is necessary for large-scale time series data
2. Describe how the Kafka/Flink streaming cold tier works
3. Describe how the native Cassandra cold tier works
4. Compare both approaches across cost, complexity, and performance
5. Understand the key technical components: Time Slices, ROCS, Backup Memtables

---

Step-by-Step Teaching

Step 1: The Core Problem — Why Tiered Storage?

Before any solution, understand the business and technical pressure.

Hot Storage (Cassandra)
├── Very fast (millisecond latency)
├── Very expensive
└── Most data is RARELY accessed after it ages

The key insight:

Not all data deserves the same storage tier

Think of it like your own files:

• Documents you edit daily → fast SSD
• Tax returns from 5 years ago → cheap external drive

Netflix faces this at petabyte scale, so even small inefficiencies cost millions.

The constraint that makes this solvable:

Old data stops being written to. It becomes immutable.

Immutability is the unlock. Once data stops changing, you can safely move, compress, and optimize it without risk of inconsistency.

---

Step 2: Foundation Concept — Time Slices

Before understanding either solution, you must understand how Netflix partitions time series data.

Timeline ──────────────────────────────────────────────►
         │  Slice A  │  Slice B  │  Slice C  │  Slice D  │
         │ (oldest)  │           │           │ (newest)  │
         │ IMMUTABLE │ IMMUTABLE │ IMMUTABLE │  ACTIVE   │

Key rules:

• Each Time Slice covers a window of time
• Each slice maps to a dedicated Cassandra table
• Once a slice crosses the write boundary, it becomes immutable
• A metadata table tracks the state of every slice

Why this matters for tiering:

Slice Age:    Old ◄─────────────────────────► New
Storage Tier: Cold Storage          Hot Storage (Cassandra)
Cost:         Cheap                 Expensive
Access Speed: Slower (~150-500ms)   Fast (<10ms)

The control plane decides when a slice is old enough to move to cold storage.

---

Step 3: Solution 1 — Kafka/Flink Streaming Cold Tier

3a. The Architecture Overview

Incoming Events
      │
      ▼
┌─────────────┐     ┌─────────────────────────────────┐
│  Cassandra  │     │  Kafka Stream                   │
│ (Hot Tier)  │     │  → Flink Workers                │
│             │     │  → Hash-partitioned buckets     │
│             │     │  → Parquet files in S3          │
└─────────────┘     └─────────────────────────────────┘

Data is written to both Cassandra and S3 simultaneously in real time.

3b. The Streaming Write Path

Why hash-partition into buckets?

Time Series IDs → Hash Function → Bucket 0, Bucket 1, ... Bucket N
                                        │
                                  Dedicated Flink Task per bucket
                                        │
                                  Batched Parquet files in S3
                                  (grouped by event time, per hour)

This prevents any single worker from being overwhelmed and enables parallelism.

Problem with raw streaming output:

Many small Parquet files = slow reads

Imagine trying to read a book where each sentence is in a separate file. You'd spend more time opening files than reading.

3c. The Compaction Job

This is the daily optimization step that fixes the small-file problem.

Before Compaction:          After Compaction:
S3/
├── hour_00_part1.parquet   S3/
├── hour_00_part2.parquet   └── day_2024_01_15_optimized.parquet
├── hour_00_part3.parquet       ├── Sorted by Time Series ID
├── hour_01_part1.parquet       ├── Row groups aligned to IDs
├── hour_01_part2.parquet       └── Bloom filters for fast lookup
└── ...

Optimizations applied during compaction:

• Sorting by Time Series ID → related data is physically adjacent
• Row group alignment → one ID = one row group = fewer reads
• Bloom filters → quickly skip row groups that don't contain your ID

Trade-off acknowledged:

Compaction takes ~12 hours on large datasets, prioritizing cost over speed

After compaction completes → control plane records the metadata → hot slice can be truncated (deleted from Cassandra).

3d. The Read Path

User Query
    │
    ▼
TimeSeries Data Plane
    │
    ├──► Hot Tier (Cassandra) ──────────────────┐
    │                                           │
    └──► Cold Tier (S3 Parquet files)           │
              │                                 │
              │  Uses compaction metadata        │
              │  to prune which files to read    │
              ▼                                 │
         At most 3 sequential S3 reads          │
              │                                 ▼
              └──────────────► Merge Results ──► Response

Performance achieved:

• p50 latency: ~150ms
• p99 latency: ~500ms
• Throughput: up to 1,250 reads/second globally

3e. Problems With This Approach

The key question this raised:

We're already backing up Cassandra to S3 for disaster recovery.
Can we just read from those backups instead?

---

Step 4: Solution 2 — Native Cassandra Cold Tier

This is the more elegant, evolved solution. It eliminates the separate streaming pipeline entirely.

4a. The Core Idea

Traditional Approach:          New Approach:
Cassandra ──► Kafka/Flink      Cassandra ──► S3 Backup (already exists!)
           ──► S3 (new copy)              ──► Read directly from backup

Netflix already backs up Cassandra to S3 for disaster recovery.
Stop paying to move data twice. Read from what already exists.

4b. Key Innovation 1: ReadOnlyCompactionStrategy (ROCS)

Before a slice can be moved to cold storage, its SSTables must be optimized.

What is an SSTable?

Cassandra stores data in immutable files called SSTables. Multiple SSTables can hold data for the same partition.

The problem ROCS solves:

Without ROCS:                    With ROCS:
SSTable 1: tokens 0-100         SSTable 1: tokens 0-33 (exact range)
SSTable 2: tokens 50-150        SSTable 2: tokens 34-66 (exact range)
SSTable 3: tokens 100-200       SSTable 3: tokens 67-100 (exact range)
→ Overlapping = must read all   → No overlap = read exactly one

ROCS process:

1. Slice crosses cold-tier boundary → writes stop → slice is immutable
2. ROCS re-compacts all SSTables once, aligning them to token ranges
3. Result: each SSTable covers a precise, non-overlapping range
4. Benefit: fewer SSTables to read = lower tail latency

Bonus: ROCS benefits ALL storage tiers, not just cold storage.

4c. Key Innovation 2: Backup Memtable

This is the most technically novel part. Understand it carefully.

Normal Cassandra read path:

Read Request
    │
    ├──► Memtable (in-memory, recent writes)
    └──► SSTables (on-disk, flushed data)

New Cassandra read path for cold slices:

Read Request
    │
    ├──► Backup Memtable (points to S3 backup!) ← NEW
    └──► SSTables (on-disk, but we DELETE these to save cost!)

How it's configured:

CREATE TABLE data_20260616_00_04 (
    ...
) WITH memtable = 'backupmemtable:timestamp=1781953200000'
--                 ↑ Custom memtable pointing to S3 backup
--                   timestamp = guaranteed full coverage point

The timestamp ensures:

• The backup fully covers the table at that point in time
• The table is guaranteed immutable — no new writes will be missed

4d. Backup Structure in S3

Understanding the S3 layout is critical for understanding how reads work efficiently.

S3 Bucket/
└── cluster-prefix/
    └── token/                    ← identifies which node
        ├── META_V2/
        │   └── snapshot.json     ← manifest listing all SSTables
        └── SST_V2/
            └── timestamp/
                └── keyspace/
                    └── table/
                        ├── na-1-big-Data.db         ← LARGE (stays in S3)
                        ├── na-1-big-Index.db         ← small (downloaded)
                        ├── na-1-big-Filter.db        ← small (downloaded)
                        ├── na-1-big-Statistics.db    ← small (downloaded)
                        ├── na-1-big-Summary.db       ← small (downloaded)
                        └── na-1-big-CompressionInfo.db ← small (downloaded)

Critical design decision:

Download the small files once. Only fetch Data.db from S3 when actually needed.

This is the key to efficiency: plan locally, fetch remotely only what's necessary.

4e. Backup Memtable Initialization State Machine

When a cold table is first opened, it goes through a reliable initialization process:

State 1: UNINITIALIZED
    │
    ▼
State 2: DOWNLOADING INDEX FILES
    │  (pulls Index, Filter, Statistics, Summary, CompressionInfo from S3)
    │  (resilience: retries, checksums, parallel downloads)
    │
    ▼
State 3: READY TO SERVE READS
    │
    ▼
State 4: ON READ REQUEST
    │  1. Check bloom filter (local) → skip if key not present
    │  2. Use index (local) → find exact byte offset in Data.db
    │  3. Fetch only needed chunks from Data.db (S3)
    │  4. Decompress and return data
    ▼
Response returned to user

Performance optimizations on reads:

• Parallelized S3 reads across requests, SSTables, and chunks
• Shared chunk cache deduplicates fetches
• Prefetching so chunks are ready before needed

4f. Verification Before Activation

Before switching a slice to cold storage, the control plane verifies two conditions across the entire ring:

Condition 1: ROCS compaction is complete on all nodes

SELECT * FROM netflix_views.cluster_view
WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';

-- Must see empty task lists on all nodes:
-- 10.0.0.2 | []   ← no active compactions ✓
-- 10.0.0.3 | []   ← no active compactions ✓

Condition 2: Backup is fully uploaded

SELECT * FROM netflix_views.complete_backups
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';

-- Must see uploaded = True:
-- my_table | 1765211400000 | uploaded = True  ✓
-- my_table | 1765213200000 | uploaded = False ✗ (not ready yet)

Only when both conditions are True does the control plane proceed.

4g. The Full Lifecycle Automation

Time Slice crosses immutable boundary
    │
    ▼
[1] ROCS re-compaction triggered
    │
    ▼
[2] Backup triggered and uploaded to S3
    │
    ▼
[3] Control plane verifies: compaction done? backup complete?
    │
    ▼
[4] Backup Memtable activated on table
    │
    ▼
[5] On-disk SSTables deleted (cost savings realized!)
    │
    ▼
[6] Slice now served from S3 backup transparently

Each slice can be at any stage independently without affecting system availability.

4h. Handling Read Spikes — The Separate Cold Cluster

The risk:

S3 reads are CPU-intensive. A spike in cold reads could overwhelm the cluster and hurt hot path performance.

The solution:

Hot Cassandra Cluster          Cold Cassandra Cluster
(expensive hardware)           (cheap, storage-light instances)
        │                               │
        │ writes backups to S3 ────────►│ reads from same S3 backups
        │                               │
        ▼                               ▼
   Serves hot reads              Serves cold reads

Key insight:

The cold cluster must match the topology (same token ring structure) but NOT the hardware. Use cheap instances.

This isolates cold read load completely from hot path performance.

---

Step 5: Comparing Both Solutions

---

Step 6: Results and Scale

Storage managed:     > 15 PB compressed data in cold storage
Reads served:        Thousands per second per region
Attached storage:    Significantly reduced on all affected clusters
p90 latency:         ~30% improvement over Kafka/Flink approach
p99 latency:         Maintained under ~500ms

---

Summary: The Mental Model

PRINCIPLE: Match storage cost to access frequency

Old Data (rarely accessed)
    └──► Cold Tier (S3) — cheap, ~150-500ms latency

New Data (frequently accessed)  
    └──► Hot Tier (Cassandra) — expensive, <10ms latency

KEY ENABLER: Immutability
    └──► Once a Time Slice stops receiving writes,
         it can be safely optimized and moved

EVOLUTION:
    Kafka/Flink approach → Works, but complex and costly
    Native Cassandra approach → Simpler, cheaper, faster
         └──► Reuse existing backups
         └──► ROCS for SSTable optimization
         └──► Backup Memtable for transparent S3 reads

The deepest lesson here is architectural:

The best solution often reuses existing infrastructure rather than building new pipelines.
Netflix already had S3 backups. The innovation was reading from them directly.