After studying this material, you should be able to:

1. Explain why large-scale Spark optimization is a continuous engineering priority
2. Describe how Apache Gluten/Velox accelerates Spark SQL through native vectorized execution
3. Compare shuffle strategies (native vs. Celeborn columnar) and their performance tradeoffs
4. Interpret benchmark results for both Gluten/Velox and Quanton evaluations
5. Explain what storage-aware optimization means and how Quanton addresses it
6. Analyze where performance gains occur within a Hudi write pipeline

---

Step-by-Step Study Material

Step 1: Understanding the Problem — Why Optimize Spark at Scale?

The Core Challenge

Before diving into solutions, understand why this matters.

Uber's Spark Scale (Daily):
├── 100+ PB data read/written
├── 80+ PB shuffle data
├── 500,000+ applications
├── 1,000,000+ compute cores
└── 200,000+ CPU cores for data lake ingestion alone

Key Insight

At massive scale, even a 1% efficiency gain translates to thousands of CPU-hours saved daily.

This is why Uber treats Spark optimization as a continuous program, not a one-time project.

Two Major Optimization Targets Identified

---

Step 2: Understanding Native Vectorized Execution

What Is the JVM Problem?

Apache Spark runs on the Java Virtual Machine (JVM). This creates a performance ceiling because:

• JVM adds abstraction overhead between your code and the CPU
• JVM cannot efficiently use modern CPU features like SIMD

What Is SIMD?

SIMD = Single Instruction, Multiple Data

Traditional (Scalar) Processing:
CPU processes one value at a time:
[A1] → process → result
[A2] → process → result
[A3] → process → result

SIMD (Vectorized) Processing:
CPU processes multiple values simultaneously:
[A1, A2, A3, A4] → process → [R1, R2, R3, R4]

Analogy: Imagine washing dishes one at a time vs. loading an entire dishwasher rack at once. SIMD is the dishwasher.

How Apache Gluten Solves This

Without Gluten:
Spark SQL → JVM Execution → CPU (no SIMD)

With Gluten:
Spark SQL → Gluten (middle layer) → Velox (native C++ engine) → CPU (with SIMD)

Apache Gluten acts as a bridge — it intercepts Spark SQL operators and redirects them to Velox, a native C++ execution engine that can use SIMD and other hardware optimizations.

---

Step 3: Understanding the Gluten/Velox Benchmark Results

TPC-DS Benchmark

TPC-DS is an industry-standard benchmark simulating complex analytical queries.

Results Summary:
┌─────────────────┬──────────────────┬─────────────┐
│ Configuration   │ Improvement      │ Notes       │
├─────────────────┼──────────────────┼─────────────┤
│ 64 executors    │ 4.0× (best case) │ Best result │
│ 32 executors    │ 2.7× (worst case)│ Still good  │
│ Overall average │ ~3.4×            │ Consistent  │
└─────────────────┴──────────────────┴─────────────┘

The Shuffle Problem — A Critical Nuance

Not all queries benefited equally. Here's why:

The Data Format Problem:

Velox works in COLUMNAR format:
[Col1: A,B,C] [Col2: 1,2,3] [Col3: X,Y,Z]

Spark shuffle works in ROW format:
[Row1: A,1,X] [Row2: B,2,Y] [Row3: C,3,Z]

When Gluten needs to shuffle data, it must convert columnar → row → columnar, adding overhead.

Real Example from the Article:

• One specific query: Spark took 10 seconds, Gluten took 1 minute
• The conversion overhead completely erased the vectorization benefit

Solution: Celeborn Columnar Shuffle

Celeborn is an external shuffle service that supports columnar data format natively.

Native Shuffle (problematic):
Velox (columnar) → convert to rows → shuffle → convert back to columnar
                   ↑ EXPENSIVE ↑

Celeborn Columnar Shuffle (solution):
Velox (columnar) → shuffle in columnar format → Velox (columnar)
                   ↑ NO CONVERSION ↑

Result: Average improvement jumped from ~3.4× to ~3.65× with Celeborn

Production Results

---

Step 4: Understanding Storage-Aware Optimization

What Gluten/Velox Doesn't Cover

Gluten/Velox is excellent for SQL query execution, but Uber's ingestion workloads involve more:

Full Ingestion Pipeline:
1. READ existing data from Hudi tables
2. APPLY updates/inserts (merge logic)
3. WRITE new files to storage
4. MAINTAIN table metadata (compaction, indexing)

Steps 2-4 are not pure SQL — they involve Hudi-specific operations that Gluten/Velox doesn't optimize.

What Is Apache Hudi?

Apache Hudi is a table format that enables:

• Incremental data ingestion (upserts, deletes)
• ACID transactions on data lakes
• Efficient record-level updates

Uber's Hudi Scale:
- ~17,200 tables managed with Hudi
- Substantial portion of ingestion compute

What Is Quanton?

Quanton is Onehouse's storage-aware Spark engine that optimizes the full pipeline:

Quanton Optimization Layers:
┌─────────────────────────────────────────────────────┐
│ EXTRACT: Native readers for Hudi/Iceberg formats    │
│          Pipelined I/O (read while computing)       │
├─────────────────────────────────────────────────────┤
│ TRANSFORM: Native vectorized SQL operators          │
│            Query planner optimizations              │
├─────────────────────────────────────────────────────┤
│ LOAD: Optimized write/merge operations              │
│       Better file creation and indexing             │
│       Table maintenance improvements                │
└─────────────────────────────────────────────────────┘

Key advantage: No code changes required — existing Scala, PySpark, and SQL applications work as-is.

---

Step 5: Understanding the Quanton Benchmark Design

Why Benchmark Design Matters

Good benchmarks must reflect real-world conditions. Uber used:

Benchmark Design Principles:
✓ Synthetic data modeled on PRODUCTION patterns
✓ Same executor allocation for both systems (fair comparison)
✓ Isolated dedicated hardware (eliminates cloud variance)
✓ Same Spark 3.5.9 and Hudi 1.1.1 versions on both sides
✓ Two table types covering different workload patterns

The Two Test Tables

Why two tables? Insert-heavy and update-heavy workloads stress different parts of the engine. Testing both gives a complete picture.

Measurement Methodology

How latency was measured:
Start of write job → [Hudi commit instant] → Start of next write job
                     ↑ cross-checked with Spark History Server ↑

Exclusion rule:
Metadata compaction rounds (~every 6-7 commits) were EXCLUDED
from steady-state figures to isolate core engine performance

---

Step 6: Interpreting the Quanton Results

Headline Numbers

Breaking Down Where Gains Came From

This is the most analytically important part:

Fact Table Write Pipeline Analysis:
┌──────────────────────────────┬──────────┬─────────┬──────────┐
│ Phase                        │ OSS Spark│ Quanton │ Speedup  │
├──────────────────────────────┼──────────┼─────────┼──────────┤
│ Data-table write             │ 475 min  │ 61 min  │ 7.8×  ✓ │
│ Metadata-table write         │ 5.6 min  │ 2.0 min │ 2.8×  ✓ │
│ Dedup and indexing           │ 25.0 min │ 27.2 min│ Slower ✗ │
│ Metadata-table compaction    │ 25.9 min │ 25.1 min│ Flat  — │
│ Small-file partitioning      │ 1.6 min  │ 1.6 min │ Flat  — │
└──────────────────────────────┴──────────┴─────────┴──────────┘

Key Takeaways:

1. Data-table write is where Quanton shines (7.8× faster) — this was the dominant cost
2. Indexing and compaction are unaffected — these remain optimization opportunities
3. Dedup/indexing was slightly slower — a tradeoff to monitor

Lesson: Optimizing one phase doesn't eliminate costs in other phases. Always profile the full pipeline.

The Scaling Experiment

Uber tested a critical question: Can Quanton do the same work with fewer resources?

Scaling Test Results (Fact Table):
┌──────────┬──────────────┬──────────────┬─────────────────────┐
│ vCPUs    │ OSS Spark    │ Quanton      │ Observation         │
├──────────┼──────────────┼──────────────┼─────────────────────┤
│ 1,128    │ baseline     │ much faster  │ Quanton wins        │
│ 564      │ (extrapolated│ comparable   │ Quanton uses half   │
│ 141      │ 78.9 min     │ 69.8 min     │ Same time, 8× fewer │
└──────────┴──────────────┴──────────────┴─────────────────────┘

Bottom line: 141 vCPU Quanton ≈ 1,128 vCPU OSS Spark
= 89% fewer vCPU-hours per write round

Why does this happen? Task skew:

Task Skew Problem (OSS Spark at large scale):
Core 1: [████████████████████] long task
Core 2: [████] short task → IDLE waiting
Core 3: [██████] short task → IDLE waiting
         ↑ Wasted capacity ↑

Quanton reduces task duration variance:
Median task duration ~10× shorter with Quanton
→ Less idle time → Better resource utilization

---

Step 7: Synthesizing the Full Picture

Two Complementary Optimization Strategies

Uber's Optimization Stack:

┌─────────────────────────────────────────────────────────────┐
│                    SPARK APPLICATIONS                        │
├─────────────────────────────────────────────────────────────┤
│  SQL/Analytics Workloads    │  Hudi Ingestion Workloads     │
│                             │                               │
│  Gluten + Velox             │  Quanton                      │
│  ├── Native SQL operators   │  ├── Native SQL operators     │
│  ├── SIMD vectorization     │  ├── Optimized Hudi writes    │
│  └── Celeborn shuffle       │  └── Storage-aware planning  │
│                             │                               │
│  Result: ~3.5× faster       │  Result: 4.7–5.5× faster     │
└─────────────────────────────┴───────────────────────────────┘

Lessons Learned Summary

---

Quick Knowledge Check

Test your understanding with these questions:

1. Why can't standard Spark use SIMD instructions efficiently?

   Because Spark runs on the JVM, which abstracts away direct CPU instruction control

2. What problem does Celeborn solve that native Gluten shuffle cannot?

   It eliminates the columnar-to-row-to-columnar conversion during shuffle

3. Why were metadata compaction rounds excluded from Quanton benchmarks?

   To isolate core engine write performance from table maintenance overhead

4. Quanton used 141 vCPUs vs OSS Spark's 1,128 vCPUs for similar completion time. What does this mean practically?

   89% fewer vCPU-hours — the same work costs ~8× less compute

5. Which Hudi pipeline phase showed NO improvement with Quanton?

   Metadata-table compaction and small-file partitioning remained flat