After studying this material, you should be able to:

1. Explain how deep learning models are interpreted using human-understandable concepts
2. Distinguish between atomic concepts (Network Dissection) and logical forms (Compositional Explanations)
3. Understand the limitations of existing evaluation methods for compositional explanations
4. Define and apply Detection Accuracy as an evaluation metric
5. Interpret what Detection Accuracy reveals about neural network units

---

Step-by-Step Study Material

Step 1: Why Do We Need to Explain Deep Learning Models?

The Core Problem

Deep learning models are extremely powerful but function as "black boxes" — they produce outputs without clearly showing why or how.

Input Image → [Black Box Neural Network] → Output: "Cat"
                        ↑
              What happened in here?

Why This Matters

• We cannot trust what we don't understand
• We cannot debug errors effectively
• We cannot verify the model learned the right things

Solution Approach

Researchers try to explain models using human-understandable concepts

💡 Key Idea: Instead of looking at raw numbers inside the network, we map network behavior to concepts humans already understand (e.g., "this unit detects eyes")

---

Step 2: Understanding Neural Network Units

What is a Unit?

A neural network is organized in layers, and each layer contains units (also called neurons or filters).

Layer 1 Units    Layer 2 Units    Layer 3 Units
[Unit 1]  →      [Unit 4]  →      [Unit 7]
[Unit 2]  →      [Unit 5]  →      [Unit 8]
[Unit 3]  →      [Unit 6]  →      [Unit 9]

What Do Units Do?

Each unit activates (responds strongly) when it detects certain patterns in the input.

💡 Goal of Explanation Methods: Assign a meaningful label/concept to each unit that describes what it detects

---

Step 3: Two Approaches to Explaining Units

Approach 1: Network Dissection (Atomic Concepts)

Atomic = single, indivisible concept

Each unit is assigned one simple concept label:

Unit 47 → "eye"
Unit 83 → "wheel"
Unit 12 → "stripe"

Limitation: Reality is more complex. A unit might detect something that cannot be captured by a single word.

---

Approach 2: Compositional Explanations (Logical Forms)

Instead of one word, units are explained using logical combinations of concepts:

Unit 47 → "eye AND (round OR oval)"
Unit 83 → "wheel AND NOT bicycle"
Unit 12 → "stripe AND (horizontal OR diagonal)"

These are called logical forms and use operators like:

• AND — both concepts must be present
• OR — either concept is present
• NOT — concept must be absent

Comparison Table

💡 Intuition: Compositional explanations are like the difference between describing someone as "tall" vs. "tall AND dark-haired AND wearing glasses"

---

Step 4: The Problem with Existing Evaluation

How Were Explanations Evaluated Before?

The most common metric was Intersection over Union (IoU):

         Area where Unit Activates ∩ Area of Concept
IoU =  ─────────────────────────────────────────────
         Area where Unit Activates ∪ Area of Concept

In simple terms:

• How much does the unit's activation overlap with where the concept appears?

The Critical Problems

Problem 1: Circular Evaluation

Search Process: "Find the best explanation by maximizing IoU"
Evaluation:     "Score the explanation using IoU"

⚠️ You are evaluating with the same metric you optimized — this is like grading your own exam you wrote yourself

Problem 2: Hyper-parameter Dependency

• Compositional explanations have a length parameter (how many concepts to combine)
• Length 1: "eye"
• Length 2: "eye AND round"
• Length 3: "eye AND round AND dark"

Question: When do you stop adding concepts? Problem: This requires manually tuning a hyper-parameter — a subjective choice

Problem 3: Unfair Comparison

• Longer explanations almost always score better on IoU
• This makes it impossible to fairly compare explanations of different lengths

---

Step 5: The Proposed Solution — Detection Accuracy

Core Idea

Instead of measuring overlap, measure consistency of detection:

"How accurately does a unit detect the presence or absence of its assigned concept across many images?"

How Detection Accuracy Works

Step 1: Take a unit with an assigned explanation (e.g., Unit 47 → "eye AND round")

Step 2: For a set of images, record:

• Does the unit activate strongly? (Yes/No)
• Does the image actually contain the concept "eye AND round"? (Yes/No)

Step 3: Compare these across all images:

Image 1: Unit activates=YES, Concept present=YES → ✓ Correct
Image 2: Unit activates=NO,  Concept present=NO  → ✓ Correct
Image 3: Unit activates=YES, Concept present=NO  → ✗ Wrong
Image 4: Unit activates=NO,  Concept present=YES → ✗ Wrong

Step 4: Calculate accuracy:

Detection Accuracy = (Number of Correct Predictions) / (Total Images)

Visual Representation

         Concept Present?
              YES    NO
Unit      YES  ✓      ✗
Activates? NO  ✗      ✓

Detection Accuracy = (✓ + ✓) / Total

---

Step 6: Three Key Advantages of Detection Accuracy

Advantage 1: Fair Comparison Across Explanation Lengths

Detection Accuracy does not automatically reward longer explanations.

Length 1: "eye"              → DA = 0.78
Length 2: "eye AND round"    → DA = 0.85
Length 3: "eye AND round AND dark" → DA = 0.83

You can now meaningfully compare these scores

---

Advantage 2: Automatic Stopping Criterion

Detection Accuracy can tell you when to stop adding concepts:

Start:    DA = 0.60  (too simple, keep going)
Length 2: DA = 0.78  (improving, keep going)
Length 3: DA = 0.85  (improving, keep going)
Length 4: DA = 0.84  (stopped improving → STOP HERE)

💡 This eliminates the need for a manually-tuned length hyper-parameter

---

Advantage 3: Discovering Specialized Units

This is the most intellectually interesting finding:

Observation: Some units have:

• A short explanation (length 1) that is a general concept
• A longer explanation that is more specific

Example:

Unit 99:
  Length 1 explanation: "face"          DA = 0.91
  Length 3 explanation: "face AND smiling AND female"  DA = 0.89

Interpretation: The unit is specialized — its simple label "face" is actually an abstraction of a more specific pattern it truly detects

These are called "specialized units" — their atomic explanation is a perceptual abstraction of their true, more complex behavior

---

Step 7: Putting It All Together

The Full Picture

PROBLEM: How do we explain what neural network units detect?

METHOD 1 (Network Dissection):
Unit → Single Concept ("eye")
[Simple but limited]

METHOD 2 (Compositional Explanations):
Unit → Logical Form ("eye AND round AND dark")
[More expressive but harder to evaluate]

OLD EVALUATION (IoU):
- Circular (same metric as search)
- Requires hyper-parameter tuning
- Cannot fairly compare lengths

NEW EVALUATION (Detection Accuracy):
- Independent of search process
- Works as stopping criterion
- Reveals specialized units
- Fairly compares all lengths

---

Summary Table

---

Quick Self-Check Questions

1. What is the difference between an atomic concept and a logical form?
2. Why is using IoU for both search and evaluation problematic?
3. How does Detection Accuracy work as a stopping criterion?
4. What does it mean for a unit to be "specialized"?
5. Why can Detection Accuracy fairly compare explanations of different lengths?