After studying this material, you should be able to:

1. Distinguish between encoding failures and recall failures in LLMs
2. Explain the five knowledge profiles and how they diagnose factual errors
3. Interpret why recall — not encoding — is the primary bottleneck in frontier LLMs
4. Apply this framework to understand specific phenomena like the long-tail problem and the reversal curse
5. Evaluate the role of "thinking" as a recall-facilitation mechanism

---

Step-by-Step Teaching

Step 1: The Core Problem — Why Do LLMs Get Facts Wrong?

When an LLM gives you a wrong answer, there are two fundamentally different explanations:

Why does this distinction matter? These two failures require completely different solutions:

• Empty shelf → need more data, bigger model
• Lost keys → need better retrieval methods, post-training techniques

Standard accuracy metrics cannot tell these apart. You just see "wrong answer" without knowing why.

---

Step 2: Three Core Concepts — Encoding, Recall, and Recognition

Think of your own memory as an analogy:

ENCODING  →  Did you ever learn this fact?
              Example: Did you ever study the capital of Uzbekistan?

RECALL    →  Can you produce it from memory, unprompted?
              Example: "What is the capital of Uzbekistan?" → "Tashkent"

RECOGNITION → Can you identify it when shown options?
              Example: "Is it (a) Almaty (b) Tashkent (c) Bishkek?" → "b"

Key insight: Recognition is easier than recall because the answer is presented to you. If a model fails recall but passes recognition, the knowledge EXISTS — it just cannot be independently retrieved.

---

Step 3: The Five Knowledge Profiles

The researchers classify every fact into one of five states:

FACT STATE DIAGRAM

Is the fact encoded?
        │
       NO ──────────────────────────► (1) ENCODING FAILURE
        │                                 "Empty shelf"
       YES
        │
Can it be directly recalled?
        │
       YES ──────────────────────────► (3) DIRECT RECALL
        │                                  "Easily accessible"
        │
       NO
        │
Can it be recalled WITH thinking?
        │
       YES ──────────────────────────► (4) RECALL WITH THINKING
        │                                  "Accessible with effort"
        │
       NO
        │
Was it encoded at all?
        │
       YES ──────────────────────────► (2) RECALL FAILURE
        │                                  "Lost keys"
        │
       NO ──────────────────────────► (5) INFERENCE WITHOUT ENCODING
                                           "Lucky guess / reasoning"

Why is this better than simple accuracy? Simple accuracy collapses profiles 1, 2, 4, and 5 all into "wrong." Knowledge profiling separates them, giving you a diagnosis, not just a score.

---

Step 4: The WikiProfile Benchmark — How They Measured This

To test these profiles at scale, the researchers built WikiProfile:

• 2,150 facts extracted from Wikipedia
• Each fact paired with 10 different questions testing encoding, recall, and recognition
• 13 LLMs evaluated, each with and without "thinking" mode
• ~4.5 million responses generated and graded

How encoding was measured:

Show the model the original Wikipedia sentence with the key fact blanked out → Can it fill in the blank?

How recall was measured:

Ask questions in varied phrasings, from different relational directions, without any context clues

How recognition was measured:

Provide multiple-choice options → Can the model select the correct answer?

---

Step 5: The Main Finding — Recall Is the Bottleneck

Here is what the data showed for frontier models (Gemini-3-Pro, GPT-5):

ENCODING RATE:     95–98%  ← Nearly everything is stored
                   ████████████████████░░

DIRECT RECALL:     66–74%  ← Many stored facts can't be retrieved
                   ██████████████░░░░░░░░

RECALL W/THINKING: 88–89%  ← Thinking helps, but gaps remain
                   █████████████████░░░░░

The critical insight: The bottleneck is not "the model doesn't know this." The bottleneck is "the model knows it but can't access it reliably."

What happens with model scaling (Gemma 3 family)?

Scaling fills the shelves but doesn't find the lost keys.

---

Step 6: Two Specific Cases That Illustrate Recall Failures

Case A: The Long-Tail Problem (Rare Facts)

Previously, researchers thought rare facts were wrong because models never learned them. This study shows something different:

                    ENCODING GAP    RECALL GAP
Popular facts:      ████████████    ████████████████████
Rare facts:         ██████████░░    ████████░░░░░░░░░░░░
                    
Gap (Δ):            SMALL           LARGE

Reframing: Rare facts ARE encoded at nearly the same rate as popular facts. The problem is that they are harder to retrieve, likely because they appeared less frequently in training, making the retrieval pathway weaker.

Case B: The Reversal Curse

The reversal curse = model knows "A is B" but fails at "What is B given A?"

Previous interpretation: The model lacks bidirectional knowledge.

This study's refinement:

TASK TYPE          DIRECT Q    REVERSE Q    GAP
─────────────────────────────────────────────────
Recognition        High        High         SMALL
(multiple choice)

Recall             High        Lower        LARGE
(open generation)

The key dissociation: If the model recognizes the correct reverse answer when shown options, but cannot generate it freely → the knowledge exists bidirectionally. The failure is in retrieval, not in storage.

Conclusion: The reversal curse is a recall problem, not an encoding problem.

---

Step 7: "Thinking" as a Recall-Facilitation Tool

"Thinking" refers to chain-of-thought reasoning, intermediate computation steps, or thinking-optimized models.

What thinking does to recall:

Encoded facts not directly recalled:
  → With thinking: 40–65% are recovered ✓

Non-encoded facts:
  → With thinking: only 5–15% recovered ✗

This tells us something important about what "thinking" actually does: It primarily facilitates retrieval of already-stored knowledge, rather than deriving genuinely new facts through reasoning.

Thinking also narrows specific gaps:

Caveat: Thinking has a computational cost, and we don't yet know the optimal strategy for deciding when to invoke it.

---

Step 8: Putting It All Together — The Big Picture Shift

BEFORE THIS RESEARCH (common assumption):
─────────────────────────────────────────
Factual errors → Model didn't learn the fact → Need more data / bigger model

AFTER THIS RESEARCH (refined understanding):
─────────────────────────────────────────────
Factual errors in frontier LLMs
         │
         ├── ~2–5% Encoding failures → Need more data/scale
         │
         └── ~26–34% Recall failures → Need better retrieval,
                                        post-training, thinking strategies

Practical implications:

---

Summary Checklist

After this material, you should be able to confirm:

• I can explain the difference between encoding and recall failures using the shelf/keys analogy
• I can name and describe all five knowledge profiles
• I understand why standard accuracy metrics are insufficient for diagnosing factual errors
• I can explain why scaling alone doesn't solve recall failures
• I can reframe the long-tail problem and reversal curse as recall issues
• I understand that thinking primarily facilitates recall, not knowledge creation
• I can articulate the shift in focus: from knowledge acquisition → knowledge utilization