After studying this material, you should be able to:
Before understanding CausalSmith, you need to understand the motivation.
The key insight from mathematics:
LLMs recently solved famous open math problems, including one Millennium Prize Problem (Navier–Stokes)
The critical enabler was formal verification — using a language called Lean 4 to check proofs without human involvement.
Human writes proof → Lean checks it → Result: Correct or Incorrect
This raises a natural question: Can this work outside pure math?
Causal inference is the study of cause and effect relationships.
Everyday examples:
Why it makes a good testbed for automated research:
| Property | Pure Math | Causal Inference |
|---|---|---|
| Results are precise theorems | ✅ | ✅ |
| Requires real-world modeling | ❌ | ✅ |
| Formally verifiable | ✅ | ✅ (with CausalSmith) |
The dual nature — rigorous like math, but grounded in real applications — makes it an ideal proving ground.
Think of the pipeline as an assembly line with quality checkpoints.
[DISCOVERY] → ✅ Review → [FORMALIZATION] → ✅ Review → [PRESENTATION]
This stage has three sub-steps:
Natural Language → [TRANSLATE] → Lean 4 Code
→ [AUDIT] → Check faithfulness
→ [PROVE] → Verify the proof
Why all three sub-steps matter:
The verified result becomes:
This makes the abstract pipeline tangible.
Starting point: A 2024 paper by Zeng et al. proved a lower bound for estimating average treatment effects, but their estimator's upper bound didn't match it — leaving a gap.
What CausalSmith did:
Step 1 (Discovery): Agent proposes a new estimator to close the gap
→ Writes natural language proof
→ LLM reviewer approves
Step 2 (Formalization): Proof needs concentration bounds for random counts
→ These aren't in Lean's standard library (Mathlib)
→ Pipeline proves them, adds to Causalean library
→ Completes full Lean verification
Step 3 (Presentation): Becomes a paper with slides and DOI
The result: "Sharp Minimax Rates for Average Treatment Effects with Discrete Confounding under Fixed Overlap" — a paper that closes a known open gap using techniques novel to causal inference.
This is a subtle but crucial concept.
The problem:
Lean guarantees the proof is correct, but NOT that the proof proves what you claimed
Example of the danger:
Natural language claim: "Theorem holds under assumptions A, B, and C"
Lean statement (sneaky): "Theorem holds under assumptions A, B, C, and D"
↑
Extra assumption makes it easier to prove
but the result is WEAKER than claimed
The solution — Auditing:
The audit step matches the Lean statement to the natural language piece by piece:
Think of it like a contract review — every clause must match exactly.
This is what makes CausalSmith self-improving rather than just automated.
Every run (success OR failure) → stored in Run Bank
↓
Future runs screen against it
→ Avoid repeating mistakes
→ Build on successes
Key insight: Failed runs are valuable data, not wasted effort.
Run discovers reusable lemma
↓
Generalize it to most general form
↓
Verify it
↓
Add to Causalean library
↓
All future runs start from a bigger foundation
Concrete example: The concentration bounds proved for the featured result were added to Causalean. The next run on the same problem reused them directly — no re-proving needed.
The compounding effect:
Run 1: Proves from scratch (slow, expensive)
Run 100: Builds on 99 runs of accumulated knowledge (faster, cheaper)
Run 1,732: Starts with 15,000+ verified theorems in Causalean
Why a new library was needed:
What Causalean covers:
| Area | Examples |
|---|---|
| Causal graphs | DAGs, d-separation |
| Potential outcomes | Counterfactuals |
| Structural causal models | SCMs |
| Estimation | Bounds, rates |
| Experiments | Randomization |
| Panel data | Difference-in-differences |
Current size: 15,000+ Lean-verified definitions and theorems
How it was built:
Results from 1,732 runs:
Total runs: 1,732 (100%)
├── Accepted: 41 (2.4%) — Correct, proved, novel
├── Downgraded: 1,388 (80.1%) — Correct math, but incremental contribution
└── Failed: 303 (17.5%) — Proposal or math rejected
How to interpret this:
Honest limitations:
| Limitation | Why It Matters |
|---|---|
| LLM reviewers judge novelty | LLMs may miss subtle prior work |
| Only small portion manually reviewed | Overall quality unknown |
| Lean checks correctness, not importance | Interesting ≠ correct |
The human-in-the-loop solution: The interactive web page allows readers to rate papers, certify individual statements, and comment — crowdsourcing human judgment at scale.
PROBLEM: Can LLMs automate theoretical research beyond pure math?
DOMAIN: Causal inference (rigorous + real-world)
SOLUTION: CausalSmith pipeline
Discovery → Formalization (Translate+Audit+Prove) → Presentation
HONESTY: Auditing ensures Lean statements match natural language claims
IMPROVES: Run bank (learn from failures) + Causalean (accumulate knowledge)
RESULT: 1,732 runs → 41 accepted papers, including one that closes
a known open gap in minimax estimation theory
LIMITS: Novelty judgment still relies on LLMs; human review needed
at scale
The core innovation is not just automation — it is self-improving automation with formal correctness guarantees, combining the rigor of theorem provers with the creativity of large language models.