parsearena 0.1.3

Run your PDFs through Docling, Marker, MinerU and Unstructured. Compare the output side by side and see what each one does to retrieval.

Leer en español · Example viewer

Why

Public parser benchmarks score parsers on their own fixed corpus: ParseBench (625 stars, by LlamaIndex) and OmniDocBench (2,146 stars). That tells you which parser wins on their documents. It does not tell you what happens to your invoices, papers or scanned reports. Harnesses that run several parsers on your own PDFs exist only as 0-star repositories (for example ticketvipers/pdf-parser-comparison and xvalovic/document-parser-benchmark).

parsearena works on your documents. It shows each parser's output next to the original page, and it measures the thing you actually care about in a RAG pipeline: after parsing and chunking, can the retriever still find the answer?

Quickstart

The demo replays outputs recorded from a real run and computes retrieval live. No keys, no GPU, no Docker, no parser downloads.

uvx --from git+https://github.com/diegoperezg7/parsearena@v0.1.3 parsearena demo

or from a clone:

git clone https://github.com/diegoperezg7/parsearena && cd parsearena && uv run parsearena demo

It writes parsearena-demo/index.html and opens it. The first run takes about 75 s (dependencies and a 65 MB embedding model); later runs take about 2 s because computed embeddings are cached. uvx parsearena is coming to PyPI.

Then run it on your own PDFs:

parsearena run docs/*.pdf --questions questions.yaml
parsearena run docs/*.pdf --gen-questions              # questions drafted by a local Ollama model
parsearena run docs/*.pdf --parsers docling,pypdf      # any subset

Example output

Recorded on 3 Nov 2026 on an Apple M4 with 16 GB of RAM (macOS 27.2), three public PDFs (8 pages), 24 hand-written questions, recall@5 with hybrid retrieval (BM25 + BAAI/bge-small-en-v1.5, fused with reciprocal rank):

What the numbers say on these pages:

• The scanned page is where the baseline breaks. pypdf has no text to read on the NACA page and loses all 6 of its answers. The four parsers OCR it; Marker and MinerU keep all 6.
• Charts and fonts. On the Census chart page pypdf returns glyph names instead of digits (/dollar.tab/eight.tab/zero.tab...), so the chart values only survive through the table page.
• Tables survive, but not always in a retrievable shape. MinerU keeps every number of Table A-1, yet it merges several row labels into one HTML cell. With the same chunker, the label and its value end up apart: the chunk with the numbers shares no word with the question, so BM25 never retrieves it, dense retrieval ranks it 7th to 16th, and the fused list puts it 31st to 37th. The other parsers mostly keep those questions in the top 10.
• Page furniture. Docling and Marker drop the first-page footer, so "NeurIPS 2023" is gone from their output. Unstructured merges the DPO results table with the body text around it and loses its 0.36 cell.
• Speed. pypdf parses a page in 0.02 s. The model-based parsers take 6 to 21 s per page on this machine, plus 20 to 100 s to load their models.

Eight pages and 24 questions are a smoke test, not a ranking: run it on your own documents.

Open the example viewer to check each of these against the page image.

How it works

flowchart LR
    P[Your PDFs] --> R1
    subgraph envs[One uv environment per parser]
      R1[Docling] ~~~ R2[Marker] ~~~ R3[MinerU] ~~~ R4[Unstructured] ~~~ R5[pypdf]
    end
    envs -->|markdown per page, seconds, errors| O[outputs.json]
    O --> C[Same chunker for all]
    Q[questions.yaml or --gen-questions] --> E
    C --> E[BM25 + fastembed + fusion]
    E --> M[answer survives, recall@k, MRR]
    M --> V[index.html viewer]
    P -->|pypdfium2| V

1. Isolated parsers. Each parser is installed with pinned versions into its own virtual environment under ~/.cache/parsearena/envs and driven through a small runner script in a subprocess. Their torch versions never meet, their licenses stay separate, and parsearena never bundles any of them. Every parser runs with its default settings.
2. Timing. Each runner first converts a two-page warm-up PDF (one text page, one image-only page) so that model loading is reported apart. The seconds per page in the table are parsing only. Device is auto: CUDA or Apple MPS when torch sees them, CPU otherwise (--device cpu to force).
3. One chunker. Every output goes through the same paragraph packer (1,200 characters by default, padding spaces and markdown table rules removed), so differences come from the parser.
4. Retrieval. For each question, chunks from all documents are ranked with BM25, with local embeddings (fastembed, ONNX, no torch) and with their reciprocal-rank fusion. Embeddings are cached by model and text under ~/.cache/parsearena/models. A chunk is relevant if it belongs to the question's document and contains the answer string. Hybrid can score below both of its parts: fusion rewards chunks that both methods rank well, so a chunk that only one of them finds (a table cell with none of the query words, say) drops in the fused list. BM25 chunks with a score of 0 are treated as not retrieved, and tied scores share a rank.
5. Metrics. Answer survives: the answer string is still in the parser output after normalization (case, whitespace, markdown and HTML markup). Recall@k: a relevant chunk is in the top k. MRR: reciprocal rank of the first relevant chunk, cut at 10.

Configuration

Environment variables: OPENROUTER_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY (only for --gen-questions with that provider), OLLAMA_HOST, PARSEARENA_HOME (where environments live), LLAMA_CPP_BINARY, PARSEARENA_LLM_TIMEOUT (seconds per LLM call, default 300). See .env.example. parsearena parsers lists pinned versions and what is installed.

--gen-questions was tested live on 3 Nov 2026 with --provider mistral --model ministral-8b-latest on two demo PDFs: 9 questions kept out of 10 asked, all sensible and with answers found verbatim in the text, for about $0.001. Review the drafted file before trusting it: one answer was just β, too short to be a useful retrieval target. --gen-questions was also tested live with mistral/ministral-8b-latest on census-income-2023.pdf: 4 questions, all grounded, $0.0006. ministral-8b-latest is the default Mistral model.

Questions file:

questions:
  - doc: report.pdf
    q: What was the median household income in 2023?
    answer: "80,610"
  - doc: paper.pdf
    q: Which preference model makes the partition function cancel?
    answers: ["Bradley-Terry", "Bradley Terry"]

What a real run costs

Be ready for this the first time:

• Downloads. Each parser environment is 1.1 to 1.4 GB, and the first parse downloads its models from Hugging Face. On the machine below, the four parser environments plus models took about 9 (5.2 GB of environments and 3.5 GB of models) of disk.

• System tools. Unstructured's hi_res strategy needs poppler and tesseract. Marker 2.x runs its OCR model through llama.cpp when there is no NVIDIA GPU: install it (brew install llama.cpp) or point LLAMA_CPP_BINARY at a llama-server binary. parsearena warns before running a parser whose tools are missing.

• Time. Measured on an Apple M4 with 16 GB of RAM (macOS 27.2), with the models already downloaded:

  Parser	Device	Model loading (s)	Seconds per page	Slowest document
  Docling	MPS	38	11.8	scanned page, 77 s
  Marker	MPS + llama.cpp (Metal)	99	21.0	scanned page, 107 s
  MinerU	MPS	20	7.6	DPO paper (equations), 43 s for 4 pages
  Unstructured	CPU (ONNX)	22	6.2	Census, 21 s for 3 pages
  pypdf	CPU	0.1	0.02

  The whole run over the 8 demo pages took about 10 minutes, one parser at a time.

Parsers and their licenses

parsearena is MIT. It installs the parsers on your machine on demand and never redistributes them; their licenses apply to your use of them.

MinerU runs at its medium effort (local layout, OCR, table and formula models, no vision-language model) with its telemetry disabled (MINERU_TELEMETRY_DISABLED=1). parsearena itself has no telemetry.

Third-party documents

The pages were extracted unchanged, except the NACA page, which was re-rendered as an image so that it has no text layer, like a real scan. These excerpts are not covered by parsearena's MIT license. Full titles, authors, sources, licenses and the CC BY attribution are in src/parsearena/demo/pdfs/NOTICE.md, which ships in the wheel. The question set is in src/parsearena/demo/questions.yaml.

Roadmap

• Cloud parsers: LlamaParse, Mistral OCR, Azure Document Intelligence
• Vision-language model parsers (and MinerU's VLM efforts)
• Table structure metrics (TEDS)
• docx and pptx input

License

MIT. Contributions welcome, see CONTRIBUTING.md.