Old OCR text cripples language model training, and FineBooks wants to fix that at scale
Key Points
- The FineBooks project, a collaboration between Hugging Face and EleutherAI, benchmarked 14 open-source OCR models on over 2,000 historical book pages to evaluate how well they can convert scanned texts into clean training data for AI language models.
- Smaller models frequently outperformed larger ones, with the top-performing model achieving over 97 percent character accuracy at a cost of less than two U.S. dollars per thousand pages.
- While the researchers consider the output quality sufficient for AI training purposes, they note that the models remain too error-prone for use in scholarly or scientific applications.
The FineBooks project from Hugging Face and EleutherAI tested 14 open-weight OCR models on more than 2,000 pages from historical books. The best models already produce text good enough for AI training, but they aren't ready for scholarly use.
Training open-source AI language models on public-domain books means dealing with bad text. Libraries extracted those texts from scans years ago using optical character recognition, and the results are often full of errors. The Talkie project put a number on the potential damage: a language model trained on OCR text learned at only 30 percent the efficiency of one trained on human transcriptions of the same books.
FineBooks, a collaboration between Hugging Face and EleutherAI, tested whether current open-source OCR models can solve this problem. The team ran 14 open-weights models on 2,165 historical book pages and published the results as a leaderboard. The best models hit character accuracy above 97 percent at less than two dollars per thousand pages.
Millions of public-domain pages need better text recognition
When EleutherAI and its partners released Common Pile last year, the largest openly licensed training corpus to date, it contained about 300,000 public-domain books with text from older OCR runs. The FineBooks authors say reprocessing those books with better models is one of the most effective ways to improve open AI training datasets.
The project picked the Biodiversity Heritage Library (BHL) as its first target, which holds more than 300,000 digitized natural history documents totaling over 64 million pages. The BHL offers its collection as a bulk download via AWS.
Measuring OCR quality requires pages with known correct transcriptions. The team used work from the IMPACT project and BHL-Europe: between 2011 and 2012, experts transcribed six BHL volumes in English, French, German, and Latin with an error rate of about one character per 2,000. That data is available under a CC-BY license in a GitHub repository and forms the basis of the new ground-truth dataset.
Small models beat larger rivals
All 14 models are freely available and run on local hardware without an API key. The metric is Character Error Rate (CER), the share of incorrectly recognized characters. The leaderboard splits results into a "diplomatic" variant that counts modernization of archaic characters like the long s (ſ) as an error and a "reading" variant that tolerates such changes.
The leading dots.mocr model uses just 3 billion parameters, while Qwen3.5-9B scores lower despite being nearly three times as large. OvisOCR2 takes second place with only 0.9 billion parameters at 46 cents per thousand pages. Model size and OCR quality don't correlate for historical documents.
| Model | Size | Accuracy | Cost/1,000 pages |
|---|---|---|---|
| dots.mocr | 3B | 97.6% | $1.94 |
| OvisOCR2 | 0.9B | 96.9% | $0.46 |
| PaddleOCR-VL-1.6 | 1B | 96.1% | $0.34 |
| olmOCR-2 | 8.3B | 95.7% | $0.45 |
| LightOnOCR-2 | 1B | 95.1% | $0.37 |
| Qwen3.5-9B | 9.7B | 94.9% | $0.89 |
| DeepSeek-OCR | 3.3B | 93.8% | $0.37 |
The evaluation covers only Antiqua typefaces in four languages. It doesn't account for Fraktur, non-Latin scripts, or handwriting, and FineBooks is limited to single-column book pages. The team plans to reprocess about 200,000 public-domain BHL documents with one of the top models and release the text as an open dataset. New models are added to the leaderboard on a rolling basis, and the evaluation framework is openly available.
Good enough for AI training, too inaccurate for scholarship
The FineBooks authors judge results by intended use. For training language models, the top performers work well enough, they write. The models produce far fewer errors than the older pipelines, and reprocessing a collection the size of BHL is realistic at the measured costs.
Libraries face a different problem, the team notes. Their systems rely on ALTO XML, a format with word-level coordinates. The new models output Markdown or plain text without word positions, so they can't plug into existing library infrastructure.
For scholarly transcriptions, accuracy still falls short, but not because the models misread characters. They silently modernize them, swapping the long "s" or ligatures for modern equivalents. Targeted fine-tuning could fix this, the team says.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.