Step-by-Step Teaching

Step 1: The Core Problem — Why Sentence-Level Translation Fails

Imagine reading a book where every sentence is translated in isolation, with no memory of what came before.

Example in English:

"John walked in. He sat down. He looked tired."

A sentence-level system would not know who "He" refers to each time. It loses context.

Sign languages have this problem even more critically because:

• Signs use physical space to represent people and objects
• Grammar is expressed through body, face, and spatial positioning
• Meaning depends heavily on what was established earlier in conversation

✅ Key Insight: Sign language is a discourse-level language — meaning is built across sentences, not just within them.

---

Step 2: What Is "Gloss Translation"?

Before going further, understand this term:

Why gloss? It is an intermediate step before generating actual signed video. Getting the gloss right is foundational.

---

Step 3: The Three Discourse Phenomena DiscoSign Addresses

🔵 Phenomenon 1: Spatial Coreference Resolution

What it means: In ASL, when you introduce a person or object, you assign them a location in space (left, right, center). Every future reference to that entity must use the same spatial location.

Example:

• Introduce "DOCTOR" → assign to left space
• Later reference to doctor must consistently use left space
• If a system forgets this, the translation becomes incoherent

DiscoSign's role: Track entities and maintain their spatial assignments across the entire discourse

---

🟡 Phenomenon 2: Question-Answer Clauses (QACs)

What it means: ASL uses a specific grammatical structure called a pseudocleft or Question-Answer Clause to emphasize information.

English equivalent concept:

"What I want is coffee" (instead of just "I want coffee")

In ASL, this structure serves specific discourse functions — it signals focus, contrast, or topic introduction.

DiscoSign's role: Detect when QAC structures are appropriate and generate them correctly, rather than defaulting to simple sentence structures.

---

🟢 Phenomenon 3: Concept-Gloss Consistency

What it means: The same English concept should always map to the same ASL sign/gloss throughout a document.

Problem without this:

• "Doctor" → DOCTOR (sentence 1)
• "Doctor" → PHYSICIAN (sentence 5)

This inconsistency confuses Deaf readers, just like randomly switching synonyms confuses hearing readers.

DiscoSign's role: Maintain a stable mapping between English words and ASL glosses throughout the entire text.

---

Step 4: The Technical Framework — How DiscoSign Works

English Text (full discourse)
         ↓
   Modular LLM Framework
   ┌─────────────────────────────┐
   │  Module 1: Coreference      │ → Tracks spatial assignments
   │  Module 2: QAC Detection    │ → Identifies discourse structures  
   │  Module 3: Gloss Mapping    │ → Ensures consistent vocabulary
   └─────────────────────────────┘
         ↓
   ASL Gloss Translation
   (discourse-coherent output)

Why modular? Each discourse phenomenon is complex. A modular design means:

• Each problem is handled separately and specifically
• Modules can be improved independently
• The system is interpretable — you can see which module did what

Why LLM-based? Large Language Models have strong language understanding capabilities, making them suitable for:

• Understanding context across long passages
• Generating linguistically appropriate structures
• Flexible adaptation to discourse rules

---

Step 5: Why Traditional Metrics Fail — And What DiscoSign Introduces

❌ Traditional Metrics (e.g., BLEU score)

These measure word overlap between generated and reference translations.

Problem: They evaluate one sentence at a time. They cannot detect:

• Whether spatial assignments are consistent across sentences
• Whether QAC structures are used appropriately
• Whether the same concept gets different glosses

Analogy: Grading each paragraph of an essay separately without checking if the argument is coherent overall.

✅ DiscoSign's Novel Evaluation Metrics

DiscoSign introduces metrics that specifically measure:

---

Step 6: Results and Significance

What the experiments showed:

✅ DiscoSign improves discourse-level quality without sacrificing sentence-level quality.

Why this matters:

• This is the first systematic framework for discourse-level text-to-sign gloss translation
• It directly benefits the Deaf and Hard-of-Hearing (DHH) community
• It establishes a methodology others can build upon

---

Step 7: Connecting to the Broader Research Context

DiscoSign is part of a larger research direction:

DiscoSign (this paper)
    → Better gloss translation at discourse level

Bootstrapping Sign Language Annotations (2026)
    → Solving the data scarcity problem with pseudo-annotation

Non-Manual Markers Research (2025)
    → Adding facial expressions and body language to generation

Big Picture: The field is moving from simple sentence translation → full, natural, discourse-aware sign language generation that truly serves the DHH community.

---

Summary Recap