Concept 1: Open-Domain Question Answering (QA)
What it is: Open-domain QA is the task of building systems that can answer questions about any topic, not just a specific domain.
Key characteristics:
- The system must search across large knowledge sources
- No topic restrictions
- Requires understanding natural language questions
Example:
"Who directed the movie Inception?" → System searches broadly and returns "Christopher Nolan"
Concept 2: Name Ambiguity in Questions
What it is: Name ambiguity occurs when a single name refers to multiple different entities.
Why it's a problem: When a question contains an ambiguous name, there are actually multiple valid interpretations — and therefore multiple correct answers.
Example from the paper:
"Which actor from the film Heat won at least one Academy Award?"
| Problem | Explanation |
|---|---|
| Multiple films named "Heat" | Which Heat are we talking about? |
| Multiple actors per film | Each film has a large cast |
| Multiple correct answers | Different actors from different films |
Key insight: A complete answer must address all valid interpretations, not just one.
Concept 3: Multi-Hop Reasoning
What it is: Multi-hop reasoning requires connecting multiple pieces of information across several reasoning steps to reach a final answer.
Think of it as a chain:
Question → Step 1 → Step 2 → Step 3 → Answer
Example breakdown:
"Which actor from the film Heat won at least one Academy Award?"
Hop 1: Identify all films named "Heat"
↓
Hop 2: Find all actors in each film
↓
Hop 3: Check which actors won Academy Awards
↓
Final: Compile complete list of qualifying actors
Why it's hard:
- Each hop introduces potential errors
- Missing one hop = incomplete or wrong answer
- Requires gathering and integrating evidence from multiple sources
Concept 4: Answer Completeness
What it is: Answer completeness measures whether a system returns all correct answers, not just some of them.
Completeness vs. Correctness:
| Metric | Question | Example |
|---|---|---|
| Correctness | "Is this answer right?" | "Actor A won an Oscar" ✓ |
| Completeness | "Did you find ALL right answers?" | "Actor A, B, and C won Oscars" ✓✓✓ |
Why existing systems fail:
- LLMs tend to return partial answers
- They may correctly identify one actor but miss others
- Exact match scores reveal this: GPT-5 scores only 0.13 on ambiguous questions
Concept 5: Knowledge Graphs + Text Corpora (Grounded QA Generation)
What it is: The paper combines two types of knowledge sources to generate verifiable questions:
Text Corpora Knowledge Graph
(unstructured) + (structured)
↓ ↓
Context & Entity relationships
descriptions & connections
↓
Verifiable, grounded questions
Why both are needed:
| Source | Strength | Weakness |
|---|---|---|
| Text Corpora | Rich context, natural language | Unstructured, hard to verify |
| Knowledge Graph | Structured, verifiable facts | Limited natural language |
| Combined | Natural AND verifiable | — |
Concept 6: Benchmarking LLMs for QA
What it is: A benchmark is a standardized test to measure and compare model performance.
DeepAmbigQA benchmark structure:
3,600 total questions
├── 1,800 Ambiguous questions (name ambiguity + multi-hop)
└── 1,800 Non-ambiguous questions (multi-hop only)
Key findings:
| Question Type | GPT-5 Exact Match Score |
|---|---|
| Ambiguous | 0.13 (13%) |
| Non-ambiguous | 0.21 (21%) |
What this tells us:
- Even the best models struggle significantly
- Ambiguity makes the problem much harder
- There is a large gap between current capability and complete answers
Concept 7: Question Rewriting for Ambiguous Questions
What it is: A technique to reformulate ambiguous or context-dependent questions into clear, standalone questions.
From the related work:
Conversational QA decomposes the task into:
- Question Rewriting → Make the question unambiguous
- Question Answering → Answer the now-clear question
Example:
Original (ambiguous): "What did he win?"
↓ rewrite
Rewritten (clear): "What award did Actor John Smith win in 2019?"
Summary: How All Concepts Connect
Ambiguous Question
↓
[Name Ambiguity Resolution] ← Concept 2
↓
[Multi-Hop Reasoning] ← Concept 3
(across text + knowledge graph) ← Concept 5
↓
[Complete Answer Set] ← Concept 4
↓
[Evaluated by Benchmark] ← Concept 6
(DeepAmbigQA)
↓
Finding: Even GPT-5 falls far short → Need better QA systems
The core message: Current LLMs are not yet reliable at producing complete answers to questions that combine name ambiguity with multi-hop reasoning — and we now have a benchmark to measure exactly how far we need to go.