← Home

olmOCR 2 improves document OCR, achieving 82.4 points on complex layouts and enhancing adaptability.

Peter Bubenik · AllenAI (Olmo model) · · Source
olmOCR 2 improves document OCR, achieving 82.4 points on complex layouts and enhancing adaptability.

TL;DR

olmOCR 2 turns messy, real-world PDFs into reliable structured text in one pass—using “unit-test-style” correctness checks during training to hit state-of-the-art accuracy where OCR usually breaks (tables, math, multi-column pages).

Key Discoveries (12 total)

  1. olmOCR 2 (olmOCR-2-7B-1025) achieves state-of-the-art OCR for English-language digitized print documents, scoring 82.4 on olmOCR-Bench.

  2. The main leap comes from training on verifiable correctness—not just more data or bigger models, meaning the system is optimized to be right in checkable ways (not just “looks plausible”).

  3. olmOCR 2 reads complex pages in a single pass, avoiding fragile multi-stage OCR pipelines that often fail on tricky layouts.

  4. It outputs structured text directly: Markdown for document structure (headings/sections), HTML for tables, and LaTeX for math—so structure isn’t “patched in later” with rules.

  5. The model is built on Qwen2.5-VL-7B and fine-tuned on a large, diverse dataset (olmOCR-mix-1025) containing 270,000 PDF pages spanning academic papers, legal docs, historical scans, brochures, and more.

  6. The dataset was expanded with 20,000 additional difficult handwritten and typewritten pages, targeting a known pain point for OCR.

  7. They introduced “unit tests” as deterministic verifiers in olmOCR-Bench, programmatic checks like “table structure preserved” or “reading order consistent” that can automatically judge correctness.

  8. They built a synthetic document pipeline to generate training data with built-in, checkable ground truth, by re-rendering pages into clean semantic HTML and deriving exact targets and test cases from that source.

  9. This pipeline produced olmOCR-synthmix-1025: 2,186 PDF pages and 30,381 verifiable test cases at ~$0.12 per page, enabling scalable, high-quality supervision.

  10. Training uses GRPO (a reinforcement learning method) with pass/fail rewards from the unit tests, and the model generates 28 candidate outputs per document during training to learn what passes more checks.

  11. Performance gains are especially strong on classic OCR failure modes, including tables (84.9% from 72.9%), multi-column layouts (83.7% from 77.3%), and old math scans (82.3% from 79.9%).

  12. They ship a practical, adaptable release (weights, datasets, training + fine-tuning code) plus fast deployment options, including an FP8 quantized model reaching ~3,400 output tokens/sec on one H100, enabling processing ~10,000 pages for under $2.

Why This Matters

PDFs are the “locked boxes” of the digital world: full of valuable information, but hard to reliably extract—especially when the page has columns, tables, equations, or poor scan quality. When OCR gets these wrong, everything downstream suffers: search becomes unreliable, analytics break, accessibility tools misread content, and AI systems learn from corrupted text.

olmOCR 2 matters because it attacks the real problem—trustworthiness—by training the model with automatic correctness checks (unit tests) that reward faithful structure and accurate reading order, not just fluent-looking text. And because the team ships the full pipeline and fine-tuning tools, organizations can adapt it to their own document styles (forms, filings, archives) without brittle, hand-built rules—making large-scale document understanding simpler, cheaper, and more dependable.

The Research Approach

The team treated OCR like a “read-and-rebuild” problem, not a “guess the text” problem. Their goal was to make the model produce outputs that can be checked automatically for correctness, the same way software is checked with unit tests.

Here is the process, step by step, in plain terms:

  1. Start with a strong vision-language base model They built olmOCR 2 on Qwen2.5-VL-7B, a model that can look at an image (a PDF page rendered as an image) and generate text. Think of this as starting with a capable general reader, then training it to become a careful document transcriber.

  2. Teach it the “right output format” from the beginning Instead of outputting plain text and fixing structure later, the model is trained to output:

  • Markdown for headings and document structure
  • HTML for tables (so rows/columns are explicit)
  • LaTeX for math (so equations are explicit)

This matters because it forces the model to represent layout and meaning directly, not as an afterthought.

  1. Fine-tune on a large, diverse real-world dataset They fine-tuned on olmOCR-mix-1025 (270,000 PDF pages) spanning academic papers, legal documents, historical scans, brochures, and more, plus 20,000 extra difficult handwritten/typewritten pages. This is the “broad reading experience” phase: lots of variety so the model does not overfit to one document style.

  2. Add a synthetic data pipeline designed for “verifiable correctness” This is the key methodological move. They created a pipeline that produces training examples where the “ground truth” is not just text, but text + structure + checkable rules.

The pipeline works like this:

  • Sample challenging real PDF pages (tables, math, multi-column layouts, degraded scans).
  • Use Claude Sonnet 4 to analyze the page layout and re-create it as clean semantic HTML (semantic means the HTML represents what things are—like a table—rather than just how they look).
  • Iteratively refine that HTML until it visually matches the original page closely.
  • From that controlled HTML source, derive the exact target output (the Markdown/HTML/LaTeX the model should produce).
  • Also from that same source, auto-generate “unit tests” (programmatic checks) that can later score whether a model output preserved the table structure, reading order, equation content, and so on.

This produced olmOCR-synthmix-1025: 2,186 pages and 30,381 test cases at about $0.12 per page.

  1. Train with reinforcement learning using pass/fail “unit test” rewards They used GRPO (Group Relative Policy Optimization), a reinforcement learning method. In simple terms: instead of only learning from “here is the correct answer,” the model learns from “this answer passes more checks than that answer.”

Concretely:

  • For each training page, the model generates 28 candidate outputs.
  • Each candidate is run through the unit tests.
  • Candidates that pass more tests get higher reward.
  • The model updates to produce outputs that pass more tests next time.

They also add smaller “helper” rewards, like:

  • Clean termination (ending properly rather than trailing off)
  • Consistent metadata extraction (pulling useful PDF metadata reliably)
  1. Evaluate with the same unit-test framework (but on separate data) Their benchmark, olmOCR-Bench, uses the same idea: deterministic unit tests. The important methodological detail is alignment: the training objective (pass tests) matches the evaluation objective (pass tests). They also emphasize they keep train/test splits clean and do not train on benchmark documents.

How It Works in Practice

The article gives a few concrete “method in action” examples that show what these choices buy you.

Example 1: Tables that usually break OCR Traditional OCR often outputs a table as a messy stream of text, losing row/column boundaries. Here, the model is trained to output HTML tables, and the unit tests can check properties like “did the number of rows/columns match” or “did cell boundaries stay consistent.” This is reflected in the reported jump on tables: 84.9% (up from 72.9%).

Example 2: Multi-column pages and reading order Multi-column layouts fail when OCR reads across columns in the wrong order (for example, reading the top of column 1, then jumping to column 2 too early). Their unit tests explicitly check reading order consistency, so the model is rewarded for producing text in the correct sequence. This matches the improvement on multi-column documents: 83.7% (up from 77.3%).

Example 3: Old math scans Math is hard because small symbol errors change meaning. By outputting LaTeX and using verifiers that check whether equations are faithfully transcribed, the model is trained to be precise rather than “approximately similar.” They report 82.3% on old math scans (up from 79.9%).

Example 4: A historical handwriting failure case (Lincoln letter) Earlier versions repeatedly misread the date in Abraham Lincoln’s January 10, 1864 letter. olmOCR 2 reads it correctly. Methodologically, this illustrates the benefit of adding 20,000 difficult handwritten/typewritten pages and training for correctness rather than plausibility: the model becomes less likely to “smooth over” ambiguous handwriting into a common-looking but wrong answer.

What Makes This Different

Three differences from traditional OCR pipelines explain why this approach is effective.

1. Single-pass structured output instead of multi-stage pipelines Classic systems often do:

  • detect layout → segment regions → OCR each region → stitch text back together → repair tables/math with rules

Each stage can introduce errors, and errors compound. olmOCR 2 instead learns to go from page image → structured representation in one generation step. It is more like asking a skilled assistant to rewrite the page in a clean, structured format, rather than running a chain of specialized tools and hoping they agree.

2. Training on “verifiable correctness,” not just “looks right” Most OCR training is supervised by matching text strings, which can reward outputs that are fluent but subtly wrong (especially in tables/math/order). Their unit tests act like a checklist a human would use:

  • Did you preserve the table grid?
  • Did you keep the reading order?
  • Did you copy the equation exactly?

Because these checks are deterministic, the model is pushed toward outputs that are checkably faithful.

3. Synthetic data that avoids corrupted supervision A common problem in document AI is noisy labels: if the “ground truth” text is wrong, the model learns the wrong thing. Their synthetic pipeline avoids this by generating pages from a controlled semantic HTML source, then testing against what was rendered. In other words, they are not trusting a potentially flawed transcription; they are comparing the model to a known blueprint.

(“Semantic HTML” here means the source explicitly encodes structure like headings and tables, so you can reliably derive both the target output and the tests.)

Overcoming Challenges

The article hints at several practical obstacles and how the team addressed them.

Challenge 1: OCR errors are hard to score automatically in a meaningful way If you only measure character-level similarity, you miss the real failures (wrong reading order, broken tables, mangled equations).
Solution: They built olmOCR-Bench around unit-test-style verifiers that check structural and semantic properties, not just text similarity.

Challenge 2: Getting enough high-quality “correctness-supervised” training data Hand-labeling structured OCR outputs at scale is expensive and inconsistent.
Solution: The synthetic document pipeline generates pages plus 30,381 automatically derived test cases, cheaply (~$0.12/page) and with consistent ground truth.

Challenge 3: Avoiding training signal corruption If training targets contain transcription mistakes, the model can be punished for being right or rewarded for being wrong.
Solution: By controlling the source (semantic HTML) and deriving targets/tests from it, they ensure the reward is tied to what the page truly contains.

Challenge 4: Reinforcement learning can be unstable or reward the wrong behavior If rewards are vague, models learn shortcuts.
Solution: Their rewards are mostly pass/fail checks tied to specific correctness properties, plus small auxiliary rewards (clean ending, metadata consistency) that reduce messy outputs.

Challenge 5: Making it usable at scale Even a great model is hard to adopt if it is slow or hard to deploy.
Solution: They released an FP8 quantized version reaching ~3,400 output tokens/sec on one H100, with a cost claim of ~10,000 pages for under $2, plus code and deployment options (APIs and tooling) so teams can actually run and adapt it.

More to study