Evaluating AI agents' effectiveness in scientific discovery through new benchmarks.

Concept 1: The Problem — Hype Without Evidence

What's happening?

Many teams are building AI science agents — systems that claim to design experiments, write code, and produce research papers.

The core issue:

The claims are extraordinary. The evidence behind them, usually, is not.

Think of it this way:

Imagine someone claims they can cook gourmet meals, but you only ever see them describe recipes — never actually cook. You wouldn't trust the claim without seeing them in the kitchen.

Key takeaway:

Claiming capability ≠ demonstrating capability. We need rigorous ways to test what AI agents can actually do.

---

Concept 2: "Book Smarts" vs. "Street Smarts"

This is the central distinction of the entire article.

The 2022 Discovery:

• AI models scored highly on multiple-choice grade-school science exams ✅
• Those same models scored below 10% when asked to perform experiments in a virtual environment ❌

Why this matters:

Knowing a concept and being able to use it are fundamentally different skills. This gap is what the benchmarks are designed to expose.

---

Concept 3: What Is a Benchmark?

Definition:

A benchmark is a standardized test used to measure and compare the performance of AI systems.

Why benchmarks matter:

• They provide objective, repeatable measurements
• They allow fair comparison between different AI systems
• They reveal specific weaknesses, not just general impressions

The benchmarks in this article:

1. ScienceWorld (2022) — Elementary school level
2. DiscoveryWorld (2024) — College/PhD level

Important design principle:

"We tend to release benchmarks that start out being very challenging, but they become much more popular a year or two later as models and methods catch up."

Benchmarks should be ahead of current capabilities — otherwise they don't push progress forward.

---

Concept 4: ScienceWorld — Testing Basic Scientific Execution

What is it?

A text-based virtual environment where AI agents must perform elementary-school-level science experiments.

The environment includes:

• 10 interconnected locations (kitchen, workshop, greenhouse, etc.)
• ~200 types of objects that behave realistically
• 30 task types with hundreds of randomized configurations

Example task:

Instead of asking:

"What is the boiling point of water?" (multiple choice)

ScienceWorld asks:

"Here is an unknown substance, a thermometer, and a stove. Figure out its boiling point."

Why randomization matters:

Agents can't memorize solutions. They must genuinely generalize their knowledge to new situations.

Results over time:

2022: Below 10% (same models that scored 'A' on science exams)
2025: Low 80s (dramatic improvement, but still not perfect)

Key insight:

"Knowing what a melting point is turns out to be a far cry from figuring out how to measure one."

---

Concept 5: DiscoveryWorld — Testing Full Scientific Discovery

What is it?

A harder benchmark testing whether agents can conduct end-to-end scientific investigations — from forming a hypothesis to interpreting results.

The setting:

• Takes place on Planet X, a fictional space colony
• Agent plays the role of a scientist

Why fictional settings?

Tasks are set in fictional scientific contexts so agents can't fall back on prior knowledge.

This forces genuine reasoning, not just retrieval of memorized facts.

What it tests (the full scientific method):

Step 1: Form a hypothesis
Step 2: Design an experiment
Step 3: Run the experiment
Step 4: Analyze results
Step 5: Understand what was discovered

Scale:

• 120 challenge tasks
• 8 topics (proteomics, rocket science, epidemiology, etc.)
• 3 difficulty levels
• Parametric variations that change data and environment each run

How it scores:

DiscoveryWorld doesn't just ask "Did you get the right answer?" It asks:

1. Did you follow a scientific process?
2. Do you actually understand the discovery you made?
3. Or did you just get lucky?

Results:

Key insight:

"The best agents can't complete roughly 80% of DiscoveryWorld's tasks at normal and challenge difficulty."

---

Concept 6: Why These Two Benchmarks Work Together

They form a ladder of scientific capability:

ScienceWorld ──────────────────► DiscoveryWorld
(Elementary level)                (PhD level)

Re-do known experiments           Design NEW investigations
Structured tasks                  Open-ended discovery
Single steps                      Hundreds of in-game actions
Known solutions exist             Ambiguous, novel results

Together they answer:

• Can the agent execute basic science? → ScienceWorld
• Can the agent actually do science independently? → DiscoveryWorld

---

Concept 7: Progress Is Real, But the Gap Remains

The trajectory of improvement:

ScienceWorld scores:
2022 ──── <10% ──────────────────── 2025 ──── ~80%
                    (3 years of progress)

DiscoveryWorld scores (harder tasks):
Best AI: ~20%    vs.    Human scientists: ~70%

What this tells us:

• AI has made genuine, measurable progress
• But there is still a significant gap between AI and human scientists
• Progress on easier tasks doesn't automatically mean readiness for harder ones

---

Concept 8: The Bigger Stakes — Why This Matters

The ultimate goal:

"We hope that in the near future, science agents will help treat diseases, create new materials, and generate other important discoveries."

The logical argument:

If an agent fails basic science (ScienceWorld)
        ↓
It cannot do advanced science (DiscoveryWorld)
        ↓
It cannot help cure diseases or make real discoveries

The solution:

Measure progress honestly. Benchmarks like ScienceWorld and DiscoveryWorld ensure that claims about AI science agents are backed by real evidence — not just impressive-sounding demos.

---

Summary: The Big Picture

The core message: Before trusting an AI agent to do science, make it prove it can actually do science — not just talk about it.