After studying this material, you should be able to:

1. Explain what Project Lighthouse is and why it exists
2. Understand k-anonymity and p-sensitive k-anonymity as privacy models
3. Describe the Core Mondrian algorithm and its role in anonymization
4. Apply data quality measurement frameworks to anonymized datasets
5. Use the project-lighthouse-anonymize Python library at a basic level

---

Step-by-Step Teaching

Step 1: The Problem Being Solved

Why Does Project Lighthouse Exist?

Airbnb needed to answer a difficult question:

"Are some users experiencing worse outcomes based on their race?"

The challenge has two competing needs:

The solution: Collect perceived race data, use it only for disparity measurement, and never link it to individual accounts.

✅ Key Insight: You can measure group-level patterns without exposing individual identities — this is the core idea behind privacy-by-design

---

Step 2: Understanding K-Anonymity

What Is K-Anonymity?

K-anonymity is a privacy model that ensures no individual can be distinguished from at least k-1 other individuals in a dataset.

Simple Example:

Imagine this raw dataset:

Age | ZIP Code | Race
----|----------|-----
 25 | 10001    | Black
 25 | 10001    | White
 25 | 10001    | Asian

With k=3, this group is k-anonymous because:

• Any single row is indistinguishable from 2 others
• You cannot isolate one person's race from their age/ZIP alone

If k=1, a person is completely identifiable — maximum privacy risk

Why K-Anonymity?

Higher k value → More people share same attributes → Harder to identify individuals
Lower k value  → Fewer people share attributes    → Easier to identify individuals

✅ Rule of Thumb: Project Lighthouse uses k=5 as its default threshold

---

Step 3: P-Sensitive K-Anonymity

Going One Step Further

K-anonymity alone has a weakness:

Age | ZIP Code | Race
----|----------|-----
 25 | 10001    | Black
 25 | 10001    | Black
 25 | 10001    | Black
 25 | 10001    | Black
 25 | 10001    | Black

This satisfies k=5 (5 people share attributes), but everyone has the same race — the sensitive attribute is still exposed!

P-Sensitivity Fixes This

P-sensitive k-anonymity requires that within each k-anonymous group, there are at least p distinct values of the sensitive attribute (race).

Age | ZIP Code | Race
----|----------|-----
 25 | 10001    | Black    ← p=2 requires at least
 25 | 10001    | Black       2 different race values
 25 | 10001    | White    ← second distinct value ✓
 25 | 10001    | White
 25 | 10001    | Asian

✅ Project Lighthouse uses p=2, k=5 — meaning each group must have 5+ people AND 2+ distinct race values

---

Step 4: The Core Mondrian Algorithm

How Do You Actually Achieve K-Anonymity at Scale?

The Mondrian algorithm works by recursively partitioning data:

Full Dataset
     │
     ├── Partition by Age < 30
     │        └── Check: does this partition have k+ rows?
     │                   Yes → keep splitting
     │                   No  → stop, generalize this group
     │
     └── Partition by Age ≥ 30
              └── Repeat...

What "Core Mondrian" Adds

The paper extends classic Mondrian with improvements for:

• Scalability — handles large datasets efficiently
• Flexibility — works beyond basic k-anonymity requirements
• Statistical preservation — keeps data useful for analysis after anonymization

✅ Think of it like: Sorting people into groups where each group is large enough that no individual stands out, while keeping groups as meaningful as possible

---

Step 5: Measuring Data Quality

The Critical Question

After anonymization, data gets generalized (made less specific). This raises a key concern:

"Is my anonymized data still good enough to draw valid statistical conclusions?"

Three Primary Quality Metrics

The paper introduces three metrics to answer this:

Metric 1: [Coverage/Completeness]
→ How much of the original data survived anonymization?

Metric 2: [Distortion/Generalization]
→ How much were values changed or broadened?

Metric 3: [Statistical Validity]
→ Do analyses on anonymized data match original data patterns?

(Note: The article references these exist but doesn't name them explicitly — the full paper provides exact definitions)

Empirical Validation Approach

They treat quality assessment as a machine learning classification problem:

Input:  Quality metrics from anonymized dataset
Output: Will this data produce statistically valid results? (Yes/No)

Validated using: Synthetic datasets with known ground truth

✅ Why this matters: Analysts without deep privacy expertise can run a simple check and get a confident answer about data usability

---

Step 6: Using the Library

Installation

pip install project-lighthouse-anonymize

Basic Workflow

p, k = 2, 5  # Define privacy parameters

# STEP 1: Apply k-anonymity
anon_df, dq_metrics, disclosure_metrics = k_anonymize(
    logger,      # logging object
    input_df,    # your raw dataframe
    qids,        # quasi-identifying columns (e.g., age, ZIP)
    k,           # minimum group size
    {},          # additional options
    "row_id"     # unique row identifier
)

# STEP 2: Apply p-sensitive k-anonymity
sensitized_df, _, _ = p_sensitize(
    logger,
    anon_df,                      # output from step 1
    qids,
    "race",                       # sensitive attribute column
    p,                            # minimum distinct values
    k,
    sens_attr_value_to_prob       # probability weights per race value
)

# STEP 3: Check data quality
minimum_dq_met, reasons = check_dq_meets_minimum_thresholds(dq_metrics)
assert minimum_dq_met, str(reasons)  # Fails if quality is too low

The Three-Step Mental Model

Raw Data
   ↓
[k_anonymize]     → Groups rows so no individual is unique
   ↓
[p_sensitize]     → Ensures sensitive attribute diversity within groups
   ↓
[check_dq_...]    → Validates data is still analytically useful
   ↓
Safe, Usable Anonymized Data

---

Step 7: The Bigger Picture

How the Three Papers Connect

Paper 1 (2020): WHY
└── Privacy-by-design rationale
└── Choosing k-anonymity as the model

Paper 2 (New): HOW
└── Core Mondrian algorithm
└── Scalable implementation details

Paper 3 (New): HOW WELL
└── Data quality measurement framework
└── Validation methodology

Key Principles to Remember

---

Summary: What You Should Now Know

🎯 Core Takeaway: It is possible to measure discrimination at scale while protecting individual privacy — these goals are not mutually exclusive when the right technical frameworks are applied.