Sakhanda Wire
NVDA $153.63 -1.61% MSFT $506.84 -2.15% GOOGL $280.27 -1.22% META $619.77 -1.06% AMZN $256.65 -0.43%
← Back to the news

Open benchmark targets bias in PDF extraction evaluation

Datalab has released OmniExtractBench, an open benchmark for systems that turn PDFs into JSON. It combines 620 documents from four existing suites and evaluates predictions against gold files value by value. The code is available on GitHub under Apache 2.0, while the dataset is hosted on Hugging Face under CC BY 4.0. The scorer can be installed from PyPI as omni-extract-bench v0.1.7 for Python 3.11 and later.

The benchmark was designed to address possible vendor bias, opaque evaluation tools, unclear scoring and limited document variety. Its deterministic scorer flattens JSON into individual addresses and normalizes values before comparison. For tables, it pairs rows by content using the Hungarian algorithm, rather than relying only on position. Each value receives one of six verdicts, including matched, misread, unfound and fabricated. Empty strings, None and whitespace are treated as omissions, preventing optional null fields from inflating results.

Datalab evaluated 10 system configurations on the full collection. Datalab accurate recorded 93.85 accuracy, while Datalab balanced and Reducto deep_extract v2 were near 93.5. The package also supports rerunning vendor tests, although users need their own API keys and paid credits.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news