After studying this material, students should be able to:

1. Define code-switching (CS) and explain why it poses challenges for speech/NLP systems
2. Understand how pretrained speech encoders encode language information
3. Explain the concept of language-selective dimensions in hidden states
4. Compare different approaches to CS detection and their trade-offs
5. Evaluate why lightweight CS detection matters for building multilingual corpora

---

Step-by-Step Teaching

Step 1: What is Code-Switching?

Definition:

Code-switching is the alternation between two or more languages within a single utterance or conversation

Real-World Example:

"I went to the tienda yesterday, 
 pero they were cerrado"
 
 English ──────────────────────────────────
 Spanish      ↑              ↑       ↑

Why Does It Happen?

Key Insight:

Code-switching is extremely common in multilingual communities but rarely represented in training data for AI systems

---

Step 2: Why is CS a Problem for AI Systems?

The Core Challenge:

Traditional AI Pipeline:

Speech → [Monolingual Model] → Output
           ↑
    Trained on ONE language
    
Result: FAILS when speaker switches languages

Three Specific Problem Areas:

1. Automatic Speech Recognition (ASR)

Input:  "I want to comprar some milk"
Bad output: "I want to ??? some milk"
                        ↑
              Model doesn't recognize
              Spanish word

2. Language Identification

Standard LID asks: "What language is this?"
CS utterance answer: "BOTH" ← systems aren't built for this

3. Speech Translation

Input (CS):  English + Spanish mixed
System expects: ONE source language
Result: Translation errors or system failure

---

Step 3: The Data Problem

Why CS is Underrepresented:

Typical Training Corpus:

[English data ████████████████████]
[Spanish data ████████████████████]
[French data  ████████████████████]
[CS data      ██                  ] ← Very small!

The Chicken-and-Egg Problem:

Need CS data to train CS models
         ↓
Need CS models to find CS data
         ↑

Traditional Solution (and its flaw):

Step 1: Transcribe speech → text
Step 2: Detect language switches in text
Step 3: Label as CS

Problem: Requires ACCURATE multilingual transcription
         Which itself is hard without good CS models!

---

Step 4: Speech Foundation Models — The Key Tool

What Are Foundation Models?

Think of them as pre-trained general-purpose models trained on massive amounts of data

Foundation Model Training:

Massive Speech Data → [Deep Neural Network] → Learned Representations
(thousands of hours)         ↑
                    Learns general patterns
                    about speech, language,
                    acoustics, etc.

Key Models Mentioned:

What Are Hidden States?

Input Audio Frame
      ↓
[Layer 1] → hidden state h₁ (vector of numbers)
      ↓
[Layer 2] → hidden state h₂
      ↓
[Layer N] → hidden state hₙ
      ↓
Output

Each hidden state is a high-dimensional vector encoding information about the input

---

Step 5: The Core Concept — Language-Selective Dimensions

What Are Dimensions?

A hidden state vector looks like:

h = [0.2, -0.8, 1.3, 0.0, -0.5, 2.1, ...]
     dim1  dim2  dim3  dim4  dim5  dim6

The Key Discovery:

Not all dimensions carry the same information!

Full Hidden State Vector:
[0.2, -0.8, 1.3, 0.0, -0.5, 2.1, 0.9, -1.2, ...]

Language-Selective Dimensions (subset):
[      -0.8,       0.0,       2.1,      -1.2  ]
         ↑                    ↑
  Activates for         Activates for
    English               Spanish

Analogy:

Think of it like a mixing board in a recording studio:

🎚️ Mixing Board (Hidden State)

Channel 1 (Bass):      ████░░░░  ← General audio info
Channel 2 (Treble):    ██████░░  ← General audio info  
Channel 3 (English):   ████████  ← LANGUAGE SELECTIVE!
Channel 4 (Spanish):   ░░░░░░░░  ← LANGUAGE SELECTIVE!
Channel 5 (Rhythm):    █████░░░  ← General audio info

---

Step 6: The Proposed Method — Step by Step

Phase 1: Identify Language-Selective Dimensions

Input: Monolingual recordings (English only, Spanish only)

For each dimension d in hidden state:
  - Measure activation when processing English audio
  - Measure activation when processing Spanish audio
  - If activation differs significantly → LANGUAGE SELECTIVE!

Output: Set of dimensions D_English, D_Spanish

Visual:

Dimension 42:
  English audio: activation = +2.3  ↑ HIGH
  Spanish audio: activation = -0.1  ↓ LOW
  → Language-selective for English! ✓

Dimension 17:
  English audio: activation = +1.1
  Spanish audio: activation = +1.0
  → NOT language-selective ✗

Phase 2: Frame-Level Analysis

Audio Utterance (3 seconds):
|-----|-----|-----|-----|-----|-----|
  t1    t2    t3    t4    t5    t6   ← frames

For each frame, check language-selective dimensions:

t1: English dims HIGH, Spanish dims LOW  → English
t2: English dims HIGH, Spanish dims LOW  → English
t3: English dims LOW,  Spanish dims HIGH → Spanish ← SWITCH!
t4: English dims LOW,  Spanish dims HIGH → Spanish
t5: English dims HIGH, Spanish dims LOW  → English ← SWITCH!
t6: English dims HIGH, Spanish dims LOW  → English

Phase 3: Aggregate into CS Prediction

Frame-level predictions: [EN, EN, ES, ES, EN, EN]

Question: Is this utterance code-switched?

If ONLY one language detected → NOT code-switched
If BOTH languages detected    → CODE-SWITCHED ✓

Decision Rule (tuned on small CS set):
  threshold θ: minimum % of frames in second language
  
  If Spanish_frames / total_frames > θ → CS = True

---

Step 7: Why This Method is "Lightweight"

Comparison of Approaches:

Approach 1: Full Transcription Pipeline
Speech → [Large ASR Model] → Text → [Language Detector] → CS Label
Cost: HIGH (needs accurate multilingual ASR)
Data needed: LOTS

Approach 2: Train CS Classifier
Speech → [Collect CS data] → [Train model] → CS Label
Cost: MEDIUM
Data needed: LOTS of labeled CS data

Approach 3: This Paper's Method ← LIGHTWEIGHT
Speech → [Pretrained Encoder] → [Select dimensions] → CS Label
Cost: LOW
Data needed: Small CS set just for threshold tuning

Resource Requirements:

---

Step 8: Results and Evaluation

The Benchmark: CS-FLEURS

CS-FLEURS Dataset:
- 14 language pairs
- Real code-switched speech
- Ground truth labels available

Performance Comparison (Mean F1 Score):

Method                          F1 Score    
─────────────────────────────────────────
This paper (Whisper-small)      0.798  ████████████████████ ← BEST
CNN baseline                    0.776  ███████████████████
HuBERT linear baseline          0.751  ██████████████████
─────────────────────────────────────────

F1 Score: 0 = worst, 1.0 = perfect

What F1 Score Means:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Precision: Of utterances labeled CS, how many actually are CS?
Recall:    Of actual CS utterances, how many did we find?

F1 balances both concerns

---

Step 9: Broader Implications

Why This Matters for Multilingual AI:

Current State:
Large Speech Collections → [Filter for CS] → CS Training Data
                                ↑
                         This step is HARD
                         
With This Method:
Large Speech Collections → [Lightweight CS Detector] → CS Training Data
                                    ↑
                              Fast, accurate,
                              no transcription needed

The Virtuous Cycle:

Better CS Detection
        ↓
More CS Training Data Found
        ↓
Better CS Models Trained
        ↓
Better CS Detection ← (cycle continues)

---

Step 10: Connecting All Three Papers

The Research Ecosystem:

PROBLEM: Code-switching is hard for AI

Paper 1 (This paper):
"How do we FIND CS utterances in large datasets?"
Solution: Language-selective dimensions in speech encoders

Paper 2 (Progressive Refinement):
"How do we TRAIN better CS-ASR with limited data?"
Solution: Iterative pseudo-labeling

Paper 3 (End-to-End ST):
"How do we TRANSLATE CS speech?"
Solution: End-to-end speech translation models

They Form a Pipeline:

Raw Audio Data
      ↓
[Paper 1: Detect CS utterances]
      ↓
CS-labeled Data
      ↓
[Paper 2: Train CS-ASR with pseudo-labeling]
      ↓
CS Transcriptions
      ↓
[Paper 3: Translate CS speech]
      ↓
Final Translation Output

---

Summary: Key Takeaways

---

Self-Check Questions

1. What is code-switching? Give an example.
2. Why can't we simply use a standard language identifier on CS speech?
3. What makes certain hidden-state dimensions "language-selective"?
4. Why is frame-level analysis important for CS detection?
5. What advantage does this method have over transcription-based approaches?
6. What does an F1 score of 0.798 tell us about the model's performance?
7. How do the three papers in this article complement each other?