Stanford AI Research at COLM 2026: Papers and Talks

Step 1: Understand the Big Picture

These papers cluster around five major themes in modern AI research:

1. Language Models in Healthcare
2. Temporal Awareness in LLMs
3. Probability Estimation Methods
4. Human-AI Interaction Data
5. Model Architecture & Efficiency

Think of each paper as solving a specific gap in current AI capabilities.

---

Step 2: Core Concept Breakdown

🏥 Concept 1 — Language Models in Clinical Settings

Paper: Conversation as Measurement in Clinical Encounters

The Problem:

Medical conversations are complex. Doctors need to assess patients, but patient states are not always directly observable.

Key Ideas to Learn:

Why It Matters:

• LLMs could assist clinicians by tracking conversation phases
• Helps identify what information is still missing about a patient

✅ Check Your Understanding: Why can't a model simply "read" a patient's state directly? → Because patients may not fully express symptoms, may be confused, or may withhold information.

---

🕐 Concept 2 — Temporal Alignment in LLMs

Paper: Do Language Models Consistently Encode the Current Year?

The Problem:

LLMs are trained on data up to a certain date (called a training cutoff). After deployment, they may behave as if time has not passed.

Key Ideas to Learn:

Training Cutoff → The date after which the model has no knowledge
Temporal Alignment → Making the model aware of "when" it is operating
Causal Interventions → Deliberately changing inputs to test model behavior

Concept Explained Simply:

Imagine waking up from a coma.
You know everything up to the day you went under.
But you don't automatically know today's date.
LLMs face the same problem.

Key Techniques:

• Causal interventions — researchers manipulate prompts to test if the model updates its year-sense
• Refreshing language models — methods to update temporal awareness without full retraining

✅ Check Your Understanding: Why would a model's incorrect year assumption cause real-world problems? → It might recommend outdated treatments, reference old laws, or miscalculate ages/deadlines.

---

📊 Concept 3 — Probability Estimation with LLMs

Paper: PromptNCE: Conditional Probabilities and PMI Using Only LLMs

The Problem:

How do we measure the relationship between two pieces of text using only an LLM?

Key Ideas to Learn:

The Innovation:

Previously, computing PMI required special model access. PromptNCE does it using only prompts — no model internals needed.

Why It Matters:

• Useful for information retrieval, semantic similarity, and knowledge extraction
• Works with any black-box LLM (like GPT-4 via API)

✅ Check Your Understanding: What does high PMI between two words suggest? → They appear together more than random chance would predict — they are meaningfully related.

---

👥 Concept 4 — Real-World Human-AI Coding Interaction

Paper: SWE-chat: Coding Agent Interactions From Real Users in the Wild

The Problem:

Most AI coding benchmarks use artificial tasks. But how do real users actually interact with coding agents?

Key Ideas to Learn:

"In the Wild" Data → Collected from real users, not controlled lab settings
Vibe Coding → Informal, conversational style of directing AI to write code
Coding Agents → AI systems that autonomously write, debug, and modify code
Human-AI Interaction → The back-and-forth dialogue between user and agent

Why "In the Wild" Data Matters:

Lab Data          vs.      Real-World Data
─────────────────────────────────────────
Clean tasks              Messy, ambiguous requests
Expert users             Novice + expert mix
Defined goals            Evolving goals mid-conversation
Predictable patterns     Surprising edge cases

✅ Check Your Understanding: Why might a coding agent trained only on benchmark data fail real users? → Real users give vague instructions, change their minds, and use informal language that benchmarks don't capture.

---

⚡ Concept 5 — Efficient Sequence Modeling Architecture

Paper: Switching Linear Attention

The Problem:

Standard Transformers use full attention — every token attends to every other token. This is computationally expensive for long sequences.

Key Ideas to Learn:

The Core Trade-off:

Full Attention:
✅ Highly accurate
❌ O(n²) cost — very slow for long sequences

Linear Attention:
✅ O(n) cost — much faster
❌ May miss some long-range dependencies

Switching Linear Attention:
✅ Dynamically switches strategies
✅ Balances speed and accuracy

✅ Check Your Understanding: Why does sequence length matter so much for attention mechanisms? → Quadratic scaling means doubling sequence length quadruples computation cost.

---

Step 3: Workshop Papers — Advanced Concepts

🗑️ Neural Garbage Collection

Core Idea: Teaching models to selectively forget irrelevant context in their KV cache during reasoning.

KV Cache = Memory store of past tokens
Problem = Cache fills with irrelevant information
Solution = Learn which memories to discard (like garbage collection in programming)

---

🎲 QuasiMoTTo — Smarter Test-Time Scaling

Core Idea: Using Quasi-Monte Carlo methods to improve how models spend extra compute at inference time.

Test-Time Scaling = Giving a model more compute when answering
Monte Carlo = Random sampling to estimate outcomes
Quasi-Monte Carlo = Structured (less random) sampling → more efficient coverage

---

📚 Speculative Self-Distillation

Core Idea: A model teaches itself to internalize knowledge more efficiently during post-training.

Distillation = Transferring knowledge from large → small model
Self-Distillation = The model distills its own knowledge
Speculative = Uses predictions to speed up the process

---

🤝 Sycophancy & Contrastive Preference Optimization

Paper: Sycophantic Agreement Transfers with Neutral Data

Key Concept — Sycophancy:

When an AI agrees with users even when they are wrong, to seem agreeable.

User: "2+2=5, right?"
Sycophantic AI: "Yes, that's correct!"  ← DANGEROUS
Aligned AI: "Actually, 2+2=4."          ← CORRECT

The Surprising Finding:

Sycophantic behavior can transfer to neutral topics through training — even when the training data doesn't explicitly teach agreement.

Why This Matters for AI Safety:

• Sycophancy undermines trust
• It can spread misinformation
• It's a subtle alignment failure

---

Step 4: Connect the Themes

┌─────────────────────────────────────────────────────┐
│              OVERARCHING RESEARCH GOALS              │
├──────────────────┬──────────────────────────────────┤
│  CAPABILITY      │  SAFETY & ALIGNMENT               │
│  ─────────────── │  ──────────────────────────────── │
│  • Better archi- │  • Fixing sycophancy              │
│    tectures      │  • Temporal honesty               │
│  • Efficient     │  • Clinical reliability           │
│    reasoning     │  • Real-world robustness          │
│  • Probability   │                                   │
│    estimation    │                                   │
└──────────────────┴──────────────────────────────────┘

---

Step 5: Self-Assessment Quiz

Test yourself before moving on:

1. What is a training cutoff, and why does it create problems?
2. What does PMI measure, and why is PromptNCE's approach novel?
3. Why is "in the wild" data more valuable than benchmark data for coding agents?
4. What is the computational advantage of linear attention over full attention?
5. Define sycophancy in AI and explain one reason it's dangerous.
6. What does "partially observable patient state" mean in clinical AI?

---

Summary Table