After studying this material, you should be able to:

1. Explain why a centralized metrics layer is necessary at scale
2. Describe the five pillars of Metrics Board and their functions
3. Understand how metrics flow from definition to consumption
4. Recognize how structured metadata enables both human and agent-driven analytics
5. Apply the concepts of metrics governance, quality, and trust signals

---

Step-by-Step Teaching

Step 1: The Problem — Why Ad Hoc Metrics Fail

Before understanding the solution, you need to understand the pain.

Imagine this scenario:

Three different teams at a company each build their own "engagement metric." Each uses slightly different SQL. None knows the others exist. When leadership asks "How is engagement?" — they get three different answers.

This is exactly what happened at Pinterest:

Key Insight:

A metric nobody trusts is worse than no metric at all.

---

Step 2: The Core Philosophy — Define Once, Use Everywhere

The entire Metrics Board system rests on one central idea:

Define a metric ONCE → Metrics Board handles everything else

How is a metric defined?

Using a YAML file stored in Git, treated like production code:

metric_id: mb__metrics
display_name: Metrics Board metrics count
description: Count of metrics defined in MB
metric_sql: |-
  SELECT COUNT(urn) as mb_metrics, '{end_date}' as dt
  FROM query_analytics.pincat_db_latest_state_v2
  WHERE dt = '{end_date}'
temporal_columns:
  - column_name: dt
    granularity: DAY
integrations:
  superset:
    enabled: true
compute:
  schedule: 0 9 * * *   # runs daily at 9am

Why YAML in Git?

• ✅ Version controlled (track every change)
• ✅ Code reviewed (peer-checked before merging)
• ✅ Single source of truth (one place, always current)
• ✅ Auditable history (who changed what, when)

---

Step 3: The Five Pillars — Architecture Walkthrough

Think of Metrics Board as a pipeline with five stations:

[1. INVENTORY] → [2. COMPUTE] → [3. QUALITY] → [4. PUBLISHING] → [5. SERVING]
  Define it       Run it         Validate it     Distribute it     Find & use it

---

Pillar 1: Inventory — The Single Source of Truth

What it does: Stores every metric definition in one place.

Key design decision:

Should Metrics Board enforce a strict data model (semantic layer), or accept any SQL?

Pinterest chose flexible SQL over a rigid semantic layer.

Why?

• User feedback demanded flexibility
• Thousands of different tables exist — modeling each was unrealistic
• Teams could adopt faster with less friction

How users create metrics:

• Fill out a guided UI form
• System auto-generates the YAML
• System opens a Pull Request automatically
• No hand-crafting required

---

Pillar 2: Compute — Automated Pipeline Generation

What it does: Turns each metric definition into a running data pipeline.

The mechanism:

YAML Definition
      ↓
Metrics Board reads metadata
      ↓
Auto-generates an Airflow DAG (scheduled workflow)
      ↓
Compiles SQL → SparkSQL or PrestoSQL
      ↓
Analyzes SQL dependencies (using sqlglot parser)
      ↓
Waits for upstream data → Runs → Stores results

Key benefits of one DAG per metric:

• Failures are isolated (one metric failing doesn't break others)
• Schedules are individually tunable
• Backfills are self-service (owners recompute history themselves)
• Late upstream data? System auto-detects and resumes

Business impact:

What used to take days or weeks now takes 1–2 hours

---

Pillar 3: Quality — Building Trust Automatically

What it does: Validates every metric run using two complementary engines.

Engine 1: Rule-Based Checks

Explicit, human-defined expectations:

Engine 2: ML-Based Anomaly Detection (Warden)

Learns the metric's historical patterns:

• Seasonality (weekends are always lower)
• Trends (gradual growth expected)
• Normal variation ranges

Why both? Rules catch known problems. ML catches unexpected ones that no rule anticipated.

Quality Reporting:

Each metric gets a quality report containing:

• Per-check pass rates
• Time series of results
• Gaps where validation didn't run
• Annotations (comments, linked tickets)

Alert routing:

• Fires immediately when a check fails
• Routes to the owning team
• Can be snoozed to prevent alert fatigue

---

Pillar 4: Publishing — Automatic Distribution

What it does: Sends metric data to every tool that needs it — automatically.

The key insight: Turning on a new surface requires only one flag change:

integrations:
  superset:
    enabled: true   # ← flip this, metric appears in Superset
  starrocks:
    enabled: true   # ← flip this, metric appears in StarRocks

Where metrics publish:

Two-phase publishing explained:

• Phase 1 (Metadata): Metric is cataloged and available to experiments immediately after the YAML merges — even before any data is computed
• Phase 2 (Data): Actual metric values flow to BI and analytics tools after each successful run

---

Pillar 5: Serving & Discovery — Finding and Trusting Metrics

What it does: Makes metrics findable and signals which ones to trust.

Discovery:

• All metrics flow into PinCat (the data catalog)
• Users find metrics the same place they find datasets
• Rich metadata: owner, description, quality signals, certification

Trust Signal 1: Metric Grades

Unknown → C → B → A → A+
                        ↑
              Manually verified by
              Data Quality Steward

Each grade requires progressively more quality work. Consumers know exactly what they're getting.

Trust Signal 2: Dashboard Tiers

Practical effect: Tier drives three automatic behaviors — how long it lives, how fast it runs, and what documentation it must have.

---

Step 4: Agent Enablement — Why This All Matters for AI

This is where the architecture pays an unexpected dividend.

The connection:

Structured metadata + Trusted definitions + Quality signals
                    ↓
        Machine-readable, governed data
                    ↓
        Agents can reason over it reliably

How it works technically:

• Metrics Board exposes an MCP (Model Context Protocol) server
• LLM-based tools connect to it
• Agents can find metrics, read definitions, and reason about them

Example:

User asks agent: "What was our engagement last quarter?"

Agent → queries MCP server → finds certified engagement metric → reads its SQL definition → references golden queries → returns grounded, trustworthy answer

The key insight:

The work done to make metrics trustworthy for humans is exactly what makes them usable by agents.

Governance isn't just bureaucracy — it's the foundation for AI reliability.

---

Step 5: Putting It All Together — The Full Flow

Data Producer (analyst/engineer)
        ↓
  Fills out UI form
        ↓
  YAML generated + PR opened
        ↓
  Code review + merge
        ↓
  ┌─────────────────────────────────┐
  │  IMMEDIATELY (metadata phase):  │
  │  • Cataloged in PinCat          │
  │  • Available in Helium (expts)  │
  └─────────────────────────────────┘
        ↓
  Airflow DAG auto-generated
        ↓
  Scheduled run executes SQL
        ↓
  Quality checks run (rules + ML)
        ↓
  ┌─────────────────────────────────┐
  │  AFTER COMPUTE (data phase):    │
  │  • Published to Superset        │
  │  • Published to StarRocks       │
  │  • Quality report updated       │
  └─────────────────────────────────┘
        ↓
  Consumers discover via PinCat
  Agents query via MCP server
  Experiments use via Helium

---

Summary: Key Concepts to Remember

---

Self-Check Questions

1. Why did Pinterest choose flexible SQL over a strict semantic layer?
2. What are the two phases of publishing, and why does the distinction matter?
3. How do rule-based checks and ML anomaly detection complement each other?
4. What is the relationship between a metric's grade and a dashboard's tier?
5. Why does good metrics governance directly enable reliable AI agents?