ExtractBench

ExtractBench is a benchmark for schema-guided extraction from enterprise documents. Given a document and a user-defined JSON Schema, a system must return schema-valid JSON with the correct values, every record of each repeated structure, missing fields marked null rather than invented, and source evidence for each value.

A schema defines one extraction task, and all documents of that type share it: one invoice schema covers invoices from every vendor, however different each one looks. Enterprises write a new schema for almost every workflow, so a system cannot be tuned to a fixed template. It has to handle schemas and documents it has never seen. And because agents increasingly act on extracted values before anyone reviews them, one truncated schedule or one invented value becomes a wrong payment or a wrong decision. The benchmark therefore scores completeness and traceability.

The benchmark covers 370 documents (4,869 pages) across 8 business domains and 67 document types, each type with its own schema. Every document is tagged along five independent axes: task challenge, perception challenge, table structure, length, and business domain, so a low score can be traced to its cause.

Leaderboard

Models and prices reflect each provider's official documentation; each system uses its recommended configuration.

Unified value F1 — the headline metric. Every score is an unweighted mean over documents; each document counts once, whatever its length. For raw data including per-split precision and recall, cost, and latency, see leaderboard.csv. Equal displayed Overall scores are ordered by lower cost per page. The best score in each Overall, Short, Medium, and Long column is bold; the second-best distinct score is underlined.

Top 10 of 49 systems — full table in leaderboard.csv.

Grounding F1 — a field counts only when its value is accepted and it points at the right evidence: at word level the predicted box must overlap an accepted evidence box at IoU 0.5, at page level the cited page must be correct. Scored only over the documents that carry verified box ground truth.

Top 10 of 49 systems — full table in leaderboard.csv.

Quick Start

Prerequisites: Create a .env file with the API key for the extraction system you want to evaluate (see Configuration).

# Install from PyPI (pick the extras for the systems you want to run)
pip install "llama-extract-bench[runners]"    # every provider SDK
pip install "llama-extract-bench[llamaextract]" # or just one, e.g. llamaextract / openai / anthropic / google

# Or, from a checkout of this repo
uv sync --extra runners

# Quick test run (6 documents — good for trying things out)
# (drop the `uv run` prefix if you installed from PyPI)
uv run extract-bench run llamaextract_agentic --test

# Full benchmark run (replace with any pipeline name, see "Available Pipelines" below)
uv run extract-bench run llamaextract_agentic

# View interactive reports in your browser
uv run extract-bench serve llamaextract_agentic

Available Pipelines

A pipeline is an extraction system or configuration you want to evaluate. Run uv run extract-bench pipelines for the live list, or see docs/pipelines.md.

The parse and layout-detection rosters inherited from ParseBench are still registered and runnable by name; list them with extract-bench pipelines --parse, --layout, or --all. They are hidden from the default listing because this benchmark scores extraction; the two-stage extract pipelines use them internally as their parse stage.

Dataset

Hosted on HuggingFace: llamaindex/ExtractBench

The benchmark is split by document length, with one JSONL row per (document, schema) test case plus the source PDFs:

The benchmark spans 8 business domains and 67 document types: finance and fund holdings, energy-sector regulatory forms, government procurement and customs, auto valuation, supply chain, healthcare remittance, legal and bankruptcy filings, and real estate.

The dataset is automatically downloaded when you run a pipeline. To manage it manually:

# Download the full dataset
uv run extract-bench download

# Download a small test dataset (6 documents, good for trying things out)
uv run extract-bench download --test

# Check whether the dataset has been downloaded and show summary statistics
uv run extract-bench status

Metrics

ExtractBench reports one metric for value accuracy and two for grounding. We score the grounding metrics only on fields that carry verified box ground truth.

• Unified value F1 — whether the extracted values match the expected output, under one definition for scalar fields and arrays of records. Each output is flattened into cells, one per scalar field and per aligned record subfield, and a cell is correct when it matches its expected counterpart after normalization. Precision, recall, and F1 are computed over these cells per document; slices report unweighted document means.
• Word-level grounding F1 — a field is grounded-correct only when its value is accepted and its predicted box overlaps any accepted box on its evidence list, at IoU 0.5.
• Page-level grounding F1 — the same rule against the cited source page instead of the box: a field counts only when its value is accepted and the page it cites is correct.

Failed and missing documents score zero rather than being dropped, so a pipeline cannot raise its average by erroring out on the documents it finds hardest.

Usage

Running the Benchmark

The run command runs inference, evaluates against ground truth, and generates reports:

# Evaluate an extraction system on the whole benchmark
uv run extract-bench run <pipeline_name>

# Evaluate a single split only (short, medium, long)
uv run extract-bench run <pipeline_name> --group short

# Skip calling the extraction system — just re-evaluate existing results
uv run extract-bench run <pipeline_name> --skip_inference

# Control how many documents are processed in parallel
uv run extract-bench run <pipeline_name> --max_concurrent 10

# Run on the small test dataset only
uv run extract-bench run <pipeline_name> --test

Viewing & Comparing Results

# View reports in your browser (needed because browsers block PDF rendering from file:// URLs)
uv run extract-bench serve <pipeline_name>

# Compare two extraction systems side-by-side
uv run extract-bench compare <pipeline_a> <pipeline_b>

# Generate a leaderboard across all evaluated systems
uv run extract-bench leaderboard

# Leaderboard for specific systems only
uv run extract-bench leaderboard llamaextract_agentic llamaextract_cost_effective

# Run inference only (call the extraction system, don't evaluate)
uv run extract-bench inference run <pipeline_name> data/short --output_dir output

# Run evaluation only (on existing inference results)
uv run extract-bench evaluation run output/<pipeline_name> --test_cases_dir data

# Generate detailed HTML report from evaluation results
uv run extract-bench analysis generate_report --evaluation_dir ./output/<pipeline_name>

/integrate-pipeline <name> <API docs or SDK link>

Configuration

API Keys

Each pipeline calls a specific system's API. You only need the key for the system you want to evaluate. Add it to a .env file at the project root (see .env.example for the full list):

# Only add the keys you need. For example, to evaluate LlamaExtract:
LLAMA_CLOUD_API_KEY=...

# To evaluate OpenAI-based pipelines:
OPENAI_API_KEY=...

# To evaluate Anthropic-based pipelines (including claude_code_extract_*):
ANTHROPIC_API_KEY=...

# To evaluate Google-based pipelines:
GOOGLE_API_KEY=...

# Codex coding-agent pipelines authenticate the codex CLI:
CODEX_API_KEY=...

ExtractBench does not use LLM-as-a-judge; all value scoring is deterministic. Your keys only ever call the extraction system you are evaluating.

CLI Reference

Advanced subcommands: inference, evaluation, analysis, pipeline, data

output/
├── _leaderboard.html                       # Cross-pipeline leaderboard
└── <pipeline_name>/
    ├── short/
    │   ├── *.result.json                    # Inference results
    │   ├── _evaluation_report.json          # Evaluation summary
    │   ├── _evaluation_report_detailed.html # Interactive detailed report
    │   ├── _evaluation_results.csv          # Per-example CSV
    │   └── _evaluation_report.md            # Markdown summary
    ├── medium/  (same structure)
    ├── long/    (same structure)
    ├── _errors.json                         # Per-document inference failures
    └── _metadata.json                       # Run metadata

src/extract_bench/
├── cli.py                           # Fire CLI entry point
├── pipeline/cli.py                  # End-to-end pipeline orchestration
├── data/
│   ├── download.py                  # HuggingFace dataset download
│   └── cli.py                       # Data management CLI
├── inference/
│   ├── runner.py                    # Batch inference with concurrency
│   ├── pipelines/                   # Pipeline registry (extract, parse, layout)
│   └── providers/                   # Provider implementations per product type
├── evaluation/
│   ├── runner.py                    # Parallel evaluation + failure penalties
│   ├── evaluators/                  # Product-specific evaluators
│   ├── metrics/extract/             # Unified value F1, grounding, record matching
│   └── reports/                     # CSV, HTML, markdown export
├── analysis/
│   ├── detailed_report.py           # Interactive per-split HTML report
│   └── comparison.py                # Pipeline comparison
├── test_cases/
│   ├── loader.py                    # Load test cases (JSONL or sidecar .test.json)
│   └── schema.py                    # TestCase types (Extract, Parse, LayoutDetection)
└── schemas/
    ├── pipeline_io.py               # InferenceRequest, InferenceResult
    ├── evaluation.py                # EvaluationResult, EvaluationSummary
    └── product.py                   # ProductType enum

Citation

@misc{zhang2026extractbenchbenchmarkschemaguidedenterprise,
  title={ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction},
  author={Boyang Zhang and Adrian Lyjak and Eli Stewart and Zhaoqi Li and Simon Suo},
  year={2026},
  eprint={2607.29677},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.29677},
}

Links

• HuggingFace Dataset: llamaindex/ExtractBench
• Code: run-llama/ExtractBench
• ParseBench: run-llama/ParseBench — A Document Parsing Benchmark for AI Agents