After studying this material, you should be able to:

1. Understand what it means for an AI system to be "Bayesian" in its belief updating
2. Explain the concept of "information processing gap" and why it matters
3. Analyze why LLMs fail to consistently update beliefs like a rational Bayesian agent
4. Evaluate the paradox of why non-Bayesian updates sometimes outperform Bayesian ones
5. Apply these diagnostics to identify problems in LLM-powered systems

---

Step-by-Step Teaching

Step 1: The Foundation — Why Uncertain Beliefs Matter

The Core Problem

Imagine a doctor using an AI assistant. The patient has a fever. Then a test comes back positive. Then another symptom appears.

A good reasoner updates their beliefs as each new piece of evidence arrives.

This is not optional in high-stakes domains like:

• 🏥 Medicine
• ⚖️ Law
• 🔬 Science

These domains share one critical feature:

There is rarely ONE correct answer. There is a most probable answer given available evidence.

Key Insight #1

LLMs are being deployed in domains that REQUIRE rational belief updating.
The question is: Do they actually do this well?

---

Step 2: What Is Bayesian Belief Updating?

The Bayesian Framework — A Mental Model

Think of belief updating like adjusting a weather forecast:

Each new piece of evidence rationally shifts your probability estimate.

Bayes' Theorem (Conceptually)

New Belief = Prior Belief × How well evidence fits this belief
             ─────────────────────────────────────────────────
                    Overall probability of seeing this evidence

In formal notation:

$P(H|E) = \frac{P(E|H) \cdot P(H)}{P(E)}$

Where:

• P(H) = Prior belief (before evidence)
• P(E|H) = Likelihood (how probable is evidence if hypothesis is true)
• P(H|E) = Posterior belief (after seeing evidence)

Key Insight #2

A "Bayesian" system is one that updates beliefs OPTIMALLY — 
extracting exactly the right amount of information from evidence.
Not too much. Not too little.

---

Step 3: The Novel Framework — LLMs as Information Processing Rules

What the Researchers Did

The paper introduces a new way to study LLMs:

Treat an LLM not as a text generator, but as an information processing rule — a system that takes in evidence and outputs updated probability beliefs.

The Information Processing Gap

This is the paper's central concept:

Information Processing Gap = |LLM's Belief Update| − |Ideal Bayesian Update|

Visually:

IDEAL BAYESIAN PATH:
Prior Belief ──[Evidence]──→ Posterior Belief ✓

LLM ACTUAL PATH:
Prior Belief ──[Evidence]──→ Deviated Belief ✗
                                    ↑
                         This deviation = Information Processing Gap

Why This Matters

The gap tells you:

• Gap = 0 → LLM updates perfectly like a Bayesian agent
• Gap > 0 → LLM over-reacts to evidence (overconfident)
• Gap < 0 → LLM under-reacts to evidence (ignores information)

---

Step 4: How LLMs Can Incorporate Evidence — Multiple Approaches

The researchers tested multiple methods for how LLMs process evidence:

Approach A: Direct Probability Elicitation

Ask the LLM directly:
"Given this evidence, what is the probability of X?"

• Simple but relies on LLM's internal calibration
• Often produces heuristic-based answers

Approach B: Explicit Bayesian Calculation

Force the LLM to:
1. State prior probability
2. Estimate likelihood of evidence
3. Compute posterior mathematically

• Follows Bayes' theorem step by step
• Should theoretically be optimal

Approach C: Learned Heuristic Updates

LLM uses patterns learned during training to "feel out"
how much to update — without explicit calculation

• Not mathematically grounded
• Based on statistical patterns in training data

Summary Table

---

Step 5: The Surprising Finding — The Paradox

What You Would Expect

Bayesian Update → Optimal → Best Performance
Heuristic Update → Suboptimal → Worse Performance

What Actually Happened

Non-Bayesian heuristic updates often OUTPERFORMED exact Bayesian updates on real tasks.

Why? — Model Misspecification

This is the critical concept:

BAYESIAN UPDATE IS ONLY OPTIMAL IF:
Your model of the world is CORRECT

IF your model is WRONG (misspecified):
→ Optimal processing of a wrong model = Wrong answers
→ Heuristic shortcuts may accidentally compensate for model errors

Analogy to Understand This

Imagine you have a broken compass that points 10° off north.

• Bayesian approach: Trust the compass perfectly → Walk 10° off course
• Heuristic approach: "I've learned compasses feel slightly off" → Adjust intuitively → Arrive closer to destination

Key Insight #3:
LLMs have MISSPECIFIED probabilistic models of the world.
Their internal representation of reality contains errors.
Exact Bayesian updating faithfully propagates these errors.
Heuristics accidentally correct for them.

---

Step 6: What "Misspecified" Means — Deeper Understanding

Model Misspecification Defined

A model is misspecified when:

The mathematical assumptions built into the model do not match how the real world actually works.

In LLM Context

LLMs learn probability distributions from training data. These distributions may be:

The Consequence

LLM's internal world model ≠ Real world

Therefore:
Bayesian update on LLM's model ≠ Bayesian update on reality

---

Step 7: Diagnostics — Practical Application

Using the Information Processing Gap as a Diagnostic Tool

The paper shows the gap can identify specific problems in LLM systems:

Diagnostic Pattern 1: Consistent Over-updating

Gap consistently positive across many queries
→ LLM is overconfident in evidence
→ May need calibration or confidence dampening

Diagnostic Pattern 2: Consistent Under-updating

Gap consistently negative
→ LLM ignores evidence (anchors too strongly to prior)
→ May need prompting strategies that emphasize new information

Diagnostic Pattern 3: Inconsistent Gap

Gap varies wildly across similar queries
→ LLM is internally inconsistent
→ Unreliable for systematic reasoning tasks

Practical Diagnostic Workflow

Step 1: Define a hypothesis H and prior probability
Step 2: Present evidence E to LLM, record updated belief
Step 3: Calculate ideal Bayesian posterior
Step 4: Compute gap = LLM belief − Bayesian posterior
Step 5: Analyze pattern of gaps across many queries
Step 6: Identify systematic biases or inconsistencies

---

Step 8: Synthesis — The Big Picture

Connecting All Concepts

┌─────────────────────────────────────────────────────┐
│              THE COMPLETE PICTURE                   │
│                                                     │
│  LLMs have internal probabilistic world models      │
│                    ↓                                │
│  These models are MISSPECIFIED (imperfect)          │
│                    ↓                                │
│  When updating beliefs with evidence:               │
│  • Bayesian updates = optimal but on wrong model    │
│  • Heuristic updates = imperfect but self-correcting│
│                    ↓                                │
│  Information Processing Gap measures the deviation  │
│                    ↓                                │
│  Gap analysis = diagnostic tool for LLM reliability │
└─────────────────────────────────────────────────────┘

---

Quick Self-Assessment

Test your understanding:

Q1: What is the information processing gap?

The deviation between an LLM's actual belief update and the ideal Bayesian update

Q2: Why might a non-Bayesian heuristic outperform exact Bayesian updating?

Because the LLM's underlying world model is misspecified — heuristics may compensate for model errors that Bayesian updates faithfully propagate

Q3: What does a consistently positive information processing gap tell you?

The LLM is systematically over-updating — overconfident in evidence

Q4: Why does this research matter for high-stakes domains?

Medicine, law, and science require rational belief updating under uncertainty — knowing LLMs fail at this consistently is critical for safe deployment

---

Key Takeaways