New OCR benchmark targets cleaner data for language model training
The FineBooks project, a collaboration between Hugging Face and EleutherAI, evaluated 14 open-weight OCR models on 2,165 pages from historical books. The models were tested against expert transcriptions in English, French, German and Latin, using character accuracy as the main measure.
The best performer, dots.mocr, reached 97.6 percent character accuracy and an estimated cost of $1.94 per 1,000 pages. OvisOCR2 ranked second with 96.9 percent accuracy, using 0.9 billion parameters and costing $0.46 per 1,000 pages. The results showed no direct link between model size and OCR quality on historical documents.
The researchers said the strongest systems produce text suitable for language-model training and could improve large collections of public-domain books. They plan to reprocess about 200,000 Biodiversity Heritage Library documents and publish the results as an open dataset. However, the evaluation covered only single-column pages in Antiqua typefaces and did not include Fraktur, handwriting or non-Latin scripts.
The models remain unsuitable for scholarly transcription, according to the project. They can silently modernize historical characters and ligatures, while their plain-text or Markdown output lacks the word coordinates required by existing library systems.
Written by Sakhanda Wire AI from reporting by The Decoder. Facts, quotes and full context are in the original article below.