After studying this material, you should be able to:

1. Explain why production-scale testing is necessary for big data performance issues
2. Design a parallel experiment workflow using branch isolation
3. Identify common Spark memory bottlenecks and their fixes
4. Apply persistent session management for long-running jobs
5. Recognize the limits of tuning versus architectural change

---

Step-by-Step Study Material

Step 1: Why Small Data Testing Fails for Big Data Problems

The Core Problem

Standard Advice:          Reality for Big Data:
"Test on small samples"   "Bugs only appear at full scale"
        ↓                           ↓
   Fast iteration            Missing the actual bug

Key Concept: Data Skew

Critical Insight: A standard 1% sample statistically eliminates extreme long-tail cases entirely

Why This Matters

• OOM errors, shuffle spillage, and stage retries cannot be reproduced on sampled data
• The bug only exists because of disproportionate data distribution
• Therefore, every meaningful test must run against full unsampled production data

Check Your Understanding

• Why would a 1% sample miss the problematic sessions?
• What is data skew in plain terms?

---

Step 2: The Parallel Experiment Architecture

The Old Approach vs. The New Approach

OLD: Sequential Testing
─────────────────────────────────────────
Test 1 → Test 2 → Test 3 → Test 4 → Test 5
  2hr      2hr      2hr      2hr      2hr
                                   = 10 hours total

NEW: Parallel Testing
─────────────────────────────────────────
Test 1 ─┐
Test 2 ─┤
Test 3 ─┼─→ All finish together
Test 4 ─┤
Test 5 ─┘
       = 2 hours total (slowest job only)

The Enabling Mechanism: Branch Isolation

Branch Name:  pgn/cap-experiment
                    ↓
         Automatically creates:
         ┌─────────────────────┐
         │ Unique workflow     │
         │ Unique output table │
         │ Unique namespace    │
         └─────────────────────┘

Why this works:

• Each branch = one independent experiment
• No collision between concurrent runs
• No collision with teammates' work
• Schema changes update automatically — no manual SQL required

The Mental Model Shift

Old goal: Make one run faster

New goal: Run many experiments simultaneously without interference

Check Your Understanding

• What would happen without branch isolation when running parallel tests?
• How does total execution time change when parallelizing 5 two-hour jobs?

---

Step 3: Persistent Sessions — Never Losing Your Work

The Problem

Without Persistence:
Laptop sleeps → SSH drops → Multi-hour Spark job DIES
                                        ↓
                              Hours of work lost

The Solution: tmux on Cloud

Your Laptop          Cloud Workbench
    │                      │
    │   SSH connection      │
    ├──────────────────────►│
    │                       │ ← tmux session lives HERE
    │                       │   (not on your laptop)
    │
  Close laptop
    │
    │   Reconnect later     │
    ├──────────────────────►│
    │                       │ ← Session still running!

The Complete Local Setup

Key Principle: Your laptop is disposable. The cloud does the work.

Check Your Understanding

• What is the difference between running tmux locally versus on the cloud server?
• Why does this setup enable working during travel?

---

Step 4: Spark Memory Bottlenecks — The Root Causes

Bottleneck 1: The collect_list Problem

collect_list behavior:
─────────────────────────────────────────
Event 1 ─┐
Event 2 ─┤
Event 3 ─┼─→ Entire bag buffered in RAM → OOM risk
  ...    ─┤
Event N ─┘

Problem: Memory scales with INDIVIDUAL session size,
         not total dataset size

What inflated each event's weight:

• Unnecessary fields included in the event bag
• Boxed data types (e.g., Integer instead of int) using extra memory

The Fix:

• Remove unnecessary fields → lighter events
• Use primitive types → less memory per event

Takeaway: Reduce individual event weight rather than adding more cluster memory

---

Bottleneck 2: Broadcast Join Memory Accumulation

Broadcast Join:
─────────────────────────────────────────
Small table → copied to ALL executors → fast joins
                    ↓
         BUT: memory stays pinned
                    ↓
         Multiple broadcasts accumulate
                    ↓
              Memory pressure builds

The Surgical Fix:

Before: Disable broadcast joins globally
        → Hurts query performance everywhere

After:  Lower only AQE's RUNTIME threshold
        spark.sql.adaptive.autoBroadcastJoinThreshold = 10MB
        → Freed ~3.1 GB executor memory
        → Initial query planning unaffected

Takeaway: Watch accumulated broadcasts in your execution plan, not just individual dataset sizes

---

Bottleneck 3: The Window Function Trap

WRONG approach:
─────────────────────────────────────────
All events → Window buffers ENTIRE session → row_number() → filter top N
                    ↑
              OOM happens HERE
              (before filtering even occurs)

CORRECT approach:
─────────────────────────────────────────
All events → Cap events FIRST → Window buffers smaller session → safe

The Lesson Visualized:

Filtering AFTER buffering:    Filtering BEFORE buffering:
┌─────────────────────┐       ┌──────────┐
│ 50,000 events       │       │ 50,000   │
│ buffered in memory  │  vs   │ → cap    │
│ THEN filter to 100  │       │ → 100    │
│ → OOM               │       │ buffered │
└─────────────────────┘       └──────────┘

Takeaway: Always test fixes against worst-case input skew, not just average load

---

Step 5: Tracking Experiments — Avoiding Chaos

The Problem at Scale

5 parallel runs × dozens of iterations =
Hundreds of combinations of:
  - JAR versions
  - Config settings  
  - Memory spill results
  - Stage failure counts
              ↓
    Easy to lose track of what you tried

The Solution: Automated Logging

After every run, the AI agent automatically appended:

Two-document system:

Document 1: Chronological run log
→ Full history, every result

Document 2: Symptom-to-fix lookup table
→ Distilled findings for teammates
→ "If you see X, try Y"

Key Principle: When running many parallel tests, losing track of results is a bigger risk than job failure

---

Step 6: The Four Fixes That Shipped

Problem Identified → Isolated in one branch → Validated → Shipped

Fix 1: Slim event bag (remove unnecessary fields)
Fix 2: Unbox primitive types (int vs Integer)  
Fix 3: Lower AQE broadcast threshold to 10MB
Fix 4: Move truncation before Window buffering

Each fix was independently validated before merging — the branch isolation made this possible.

---

Step 7: The Bigger Lesson — Tuning Has a Ceiling

Current state after fixes:
─────────────────────────────────────────
Pipeline runs cleanly ✓
Memory pressure reduced ✓

Future state (inevitable):
─────────────────────────────────────────
Upstream adds new event attributes
        ↓
Each event gets heavier again
        ↓
Memory pressure returns
        ↓
WITHOUT changing a single line of your code

The Fundamental Tension

Core Takeaway: If your data model allows unbounded row growth, tuning alone cannot save you permanently. Eventually the architecture must change.

---

Summary: The Complete Mental Model

PROBLEM: Big data bugs only appear at full scale
         ↓
SOLUTION LAYER 1: Branch isolation → parallel experiments
         ↓
SOLUTION LAYER 2: tmux on cloud → persistent sessions
         ↓
SOLUTION LAYER 3: AI agent → operational tracking
         ↓
RESULT: 5 experiment cycles in 24 hours instead of 5 days
         ↓
FINDINGS: collect_list weight + broadcast accumulation + Window ordering
         ↓
LESSON: Tuning buys time. Architecture determines ceiling.

---

Self-Assessment Questions

1. Why can't you reproduce data skew bugs on a 1% sample?
2. How does branch isolation prevent experiment collisions?
3. What is the mathematical advantage of parallel vs sequential testing?
4. Why does collect_list cause OOM based on individual session size rather than total data volume?
5. What is the difference between compile-time and runtime broadcast thresholds?
6. Why did the Window function cap fail at peak load but pass off-peak tests?
7. What is the difference between tuning a pipeline and redesigning its architecture?