See every page side by side: jandira-tech.github.io/neurotic_docx_bench
DOCX to PDF across engines, and redlines against Word (/redlines/), scored per page.
Navigator: DOCX to PDF · redlines vs Word (Jubarte, Docxodus, SuperDoc; redlines, accepted, rejected) · speed

neurotic_docx_bench

Pixel scores of DOCX tools against Microsoft Word oracles. Neurotic: it refuses to round, to drop failures, or to trust a renderer it has not pinned.

Who maintains this

Maintained by Arthur Souza Rodrigues (Jandira Technologies), who also writes Jubarte, one of the ranked tools. Jubarte rows are marked † and follow the same rules. Every place the harness touches another tool's code is listed in docs/VENDOR_NOTES.md.

Scores	0 to 100 per document
Redline oracle	Word renders both sides: Word's render of the candidate DOCX against Word's render of Word's own compare. Only Word-rendered rows are ranked; LibreOffice-rendered rows are history
DOCX to PDF oracle	SHA-pinned Word-export PDFs (pdf_accepted_word, pdf_redlines_randomized, pdf_source, pdf_source_randomized)
Second lens	docxide_metrics: docxide-pdf's own Jaccard / text-boundary metrics on the same fixtures, and the same two columns on every redline row
Stores	results/bench.jsonl (fidelity), results/converters.jsonl (DOCX to PDF), results/speed.jsonl + results/redline_speed_bench/ (speed), results/archive/ (history only)
Results	RESULTS.md (headline) and RESULTS_DETAILED.md (history, paired comparisons, methodology), both generated
Visual report	runs/<run>/report.html
Speed	docs/SPEED.md

One row per tool in every headline table: its latest eligible run in the current comparability group (same document set, renderer and scorer). No best pin, no best-of-N. Author-affiliated tools (Jubarte) are marked and follow the same rules. Compare rows only within one table.

uv run bench report            # regenerates RESULTS.md, RESULTS_DETAILED.md and the vendor table below
uv run bench report --check    # exit 1 when the published views are stale (CI)

For the ranked redline benchmarks Word renders both the candidate and the oracle DOCX, so the score compares like with like. Legacy rows rendered with LibreOffice 26.2.4.2 stay in the history; for those, re-rendering the same DOCX is byte-identical on one build (results/noise_floor.json). The oracle-identity calibration row (bench calibrate) is Word's own DOCX through the candidate pipeline and must score 100; the null-baseline row is the base document unchanged, the floor a redline tool must beat.

Ranked rows today: DOCX to PDF, the docxide metrics, redlines vs Word's compare, and accepted/rejected changes. Every other benchmark is waiting for a Word-rendered run and shows history only.

Limitations

• The score is pixel similarity to Microsoft Word's output. It has not been validated against human judgment of redline quality. A redline can be correct and still lay out differently from Word's compare. The Lens health table in RESULTS_DETAILED.md shows the pixel and functional lenses disagreeing on 1% to 29% of documents, depending on the tool.
• The oracle is one build of Word for Mac. Word-rendered rows can only be reproduced on a machine with Word; CI cannot regenerate them.
• Adapters for other tools are written by the Jubarte author. A low score can be the adapter's fault; docs/VENDOR_NOTES.md records the known cases, including adapter failures first found by a tool's maintainer.
• Failures count as zero. That mixes "the tool could not run on this file" with "the tool ran badly"; both the ITT and scored-only columns are printed for that reason.
• Corpus: DOCX files from superdoc-dev/docx-corpus, collected from the public web and made available under ODC-By 1.0. ODC-By covers the collection, not copyright in the individual files. A rights holder who wants a document removed can write to contact@arthur.law; it is removed from the fixtures and future runs.
• The sealed holdout has never been run ("no holdout runs recorded yet" in RESULTS_DETAILED.md).

Disputing a score

Open an issue. A harness fix that raises a competitor's score ships with the same urgency as one that raises Jubarte's (standing rule 3 in docs/VENDOR_NOTES.md).

Results visibility

RESULTS.md is the headline: one table per benchmark, intervals, calibration rows, and the reason for every row that is not ranked. RESULTS_DETAILED.md holds every row in the stores grouped by comparability group with its eligibility verdict, pairwise comparisons, lens-health alarms, and the methodology. Rows that predate provenance stamping, holdout-only runs and retracted runs live in results/archive/ with a manifest and appear only in the history. The three-way renderer corpus contains Word, Jubarte, and docxide-pdf PDFs for the same 398 source DOCX files.

The 2026-10-14 Wordcraft CLI cohort (400 comparisons and 400 conversions against fresh Word 16.115 oracles) is reported on its own in results/wordcraft_20261014_fresh/REPORT.md. Besides Wordcraft 0.6.1 and 0.7.0 it covers, on the same fixtures, jubarte 0.11.0 / 0.12.0, docxodus 12.6.5 / 13.0.0, folio-core 0.60.0, stemma 0.6.0 and superdoc-sdk 2.20.0 (comparisons), and LibreOffice, office2pdf, pdfitdown, rdocx, docxide-pdf, libreoffice_convert_rust and PyMuPDF Pro (conversions). Its sample and oracles differ from the stores above, so its numbers are not ranked against them.

Benchmarks

Compare vendors only within one table. LibreOffice scores and Playwright scores are separate measurements.

Benchmark	Input	Oracle
script_redlines	base.docx + next.docx → tool redline DOCX	Word redline PDF
accepted_changes	Accept all w:ins/w:del on the tool redline	Word redline with changes accepted
roundtrip	Self-diff or the tool’s roundtrip path	Original source PDF (identity = 100)
visual_rendering	Source DOCX in the vendor web editor	pdf_source
visual_redlines	Word redline DOCX in the vendor web editor	pdf_redlines_word
visual_accepted_changes	Accepted Word redline in the vendor web editor	pdf_accepted_word
docx_to_pdf	Accepted Word redline DOCX + randomized redline DOCX	SHA-pinned pdf_accepted_word + pdf_redlines_randomized
docx_to_pdf_no_redline_docs	Source DOCX + randomized source DOCX	SHA-pinned pdf_source + pdf_source_randomized
docxide_metrics	The docx_to_pdf_no_redline_docs inputs, scored with docxide-pdf's Jaccard / text-boundary metrics at 150 DPI	Same SHA-pinned pdf_source + pdf_source_randomized

visual_* loads Word's DOCX in the editor, not the tool's own redline. Generator package and editor package are separate pins.

Pins: bench.yaml. Tool identities: bench.registry.yaml (the table below is generated from it).

Tool	Role	Engine	Author-affiliated	Note
Wordcraft CLI	generator	wordcraft-cli
jubarte 0.11.0	generator	jubarte-redlines	yes
jubarte 0.12.0	generator	jubarte-redlines	yes
docxodus 12.6.5	generator	docxodus
Wordcraft CLI 0.7.0	generator	wordcraft-cli
docxodus 13.0.0	generator	docxodus
folio-core 0.60.0	generator	@stll/folio-core
stemma 0.6.0	generator	stemma
superdoc-sdk 2.20.0	generator	superdoc-sdk
docxide-pdf 0.18.4	converter	docxide-pdf
rdocx 0.16.0	converter	rdocx
office2pdf 0.8.1	converter	office2pdf (Typst)
pdfitdown 4.2.0	converter	office2pdf (Typst)		Delegates Office formats to office2pdf; same engine, listed for completeness.
libreoffice_convert_rust 0.1.0	converter	LibreOffice
PyMuPDF Pro 1.28.2	converter	pymupdf-pro		Unlicensed (Restricted Mode): converts only the first 3 pages, so it is scored on the at-most-3-page set.
jubarte (lossless)	generator	jubarte-final	yes
jubarte (ast)	generator	jubarte-final	yes
jubarte-rust	generator	jubarte-redlines	yes
jubarte-wasm	generator	jubarte-redlines	yes
docxodus	generator	docxodus
docxodus (C#)	generator	docxodus		C# build of docxodus; speed benchmark only.
folio	generator	@stll/folio-core
superdoc	generator	superdoc-sdk
redlines	generator	redlines
superdoc-redlines	generator	superdoc-redlines
stemma	generator	stemma
safe-docx	generator	safe-docx
docxodus (viewer)	editor	react-docxodus-viewer
folio (viewer)	editor	@stll/folio-react
superdoc (editor)	editor	superdoc
jubarte	converter	jubarte-redlines	yes
rdocx	converter	rdocx
office2pdf	converter	office2pdf (Typst)
pdfitdown	converter	office2pdf (Typst)		Delegates Office formats to office2pdf; same engine, listed for completeness.
doxx	converter	doxx		No PDF export; listed as not applicable, never ranked.
libreoffice_convert_rust	converter	LibreOffice
soffice	converter	LibreOffice
dxpdf	converter	dxpdf
docxide-pdf	converter	docxide-pdf
PyMuPDF Pro	converter	pymupdf-pro		Unlicensed (Restricted Mode): converts only the first 3 pages, so it is scored on the at-most-3-page set.

---

Install

Version 1.0.0 is built for PyPI as neurotic-docx-bench but not uploaded yet; until it is, install from a checkout (uv tool install ., or pip install .). macOS and Linux only (the hub download locks files with fcntl).

pip install neurotic-docx-bench            # once 1.0.0 is on PyPI; or: uv tool install neurotic-docx-bench
bench fixtures list v1                     # the fixture set, its hub commit, cases per test
bench fixtures get v1                      # 14,513 files (2.5 GB) into corpus/word, each sha256-checked
bench fixtures get v1 --test redline       # one test's files only (the corpus tables always come along)

Version 1 of the package carries fixture set v1: every file on the hub dataset arthrod/neurotic_docx_bench-fixtures whose oracle Microsoft Word made, for every test the bench runs, pinned to dataset commit 984e64d869803abea841e66364f7b82bb30b3a70:

test	a case is	cases
docx_to_pdf	a document and Word's PDF of it	2,925
redline	base and next documents, Word's compare of them and Word's PDF of the compare	3,502
accept_all	a Word compare, Word's Accept All of it and Word's PDF of the result	400
reject_all	a Word compare, Word's Reject All of it and Word's PDF of the result	400

The wheel ships the set's manifest (neurotic_docx_bench/data/fixture_sets/v1.csv.gz: test, case, state, role, path, sha256, size, git blob id), never the documents. bench fixtures get reads exactly those files at the pinned commit, checks the hub's listing against the manifest before downloading, and every downloaded file against its sha256 after. From Python: neurotic_docx_bench.fixture_sets (load, rows, files, get). LibreOffice renders, superseded Word renders and the one compare Word left without a PDF are not in v1. Generators for the competitor tools are the competitors extra (pip install 'neurotic-docx-bench[competitors]').

Quick start (from a checkout)

Python 3.14 (uv), Node 26 + pnpm, LibreOffice 26.2.4.2 on PATH.

uv sync
pnpm install --frozen-lockfile   # workspace: root pin tree + the utils/ engine fallbacks + live-bench frontend
# (only for the visual_* playwright benchmarks:) cd harness/folio-viewer && bun install

uv run bench run --only jubarte-final-lossless --limit 5
uv run bench run
uv run bench docx-to-pdf --tool jubarte --tool rdocx --tool office2pdf --tool pdfitdown --tool doxx

Each bench run: resolve tool_version → generate → render → score → append results/bench.jsonl → gate vs snapshot.

uv run bench run --only docxodus --limit 5 --no-emit
uv run bench accept-scores jubarte
uv run bench render <docx-dir> <work-dir> -b soffice
uv run bench compare <candidate-pdfs> <oracle-pdfs> --tool name

Try one fixture with your own tool

bench try scores any tool on one fixture of the tryout set (100 pairs drawn from the Word-exported corpus, corpus/tryout/tryout_100.csv) against the Word oracle, next to jubarte's precomputed output for the same pair. Nothing is written under results/.

uv run bench try list
uv run bench try run --random --task redline --tool "mytool {base} {next} -o {out}" --json out/try.json
uv run bench try run --fixture <pair_stem> --task convert --tool "mytool {input} -o {out}" --renderer soffice
uv run bench try fetch <pair_stem> --dest tryout_dl && uv run bench try run --root tryout_dl --fixture <pair_stem> --tool "..."

--renderer passthrough (default) expects the tool to write the PDF itself; soffice and word render a .docx output first. --against takes another command template or a PDF path instead of jubarte. --seed repeats a random pick. scripts/tryout_jubarte.py regenerates jubarte's outputs for the set (bench try build-set redraws it).

The Word corpus

Word is the source of truth for PDFs. Everything Word produced lives under corpus/word/, tracked in this repository (docx and PDFs included) and the one source of truth. It was built by copy (bench corpus build --origins) from Word's working folders and, with --fixtures, from the jubarte-first _fixtures folder; those folders are not part of the repository. Only what Word finished is in: docx Word could not open, blacklisted stems and the compares touching them, rejected compares and, in the sets that are Word render runs, documents Word did not render to PDF are left out and listed per set in PROVENANCE.json. A PDF whose producer is not Word (LibreOffice, a tool's own writer) is refused.

The tree is laid out by provider and by the state of the docx, read from its XML:

corpus/word/
  clean/                      docx/  pdf/  pdf_prior/
  tracking_without_comments/  docx/  pdf/  pdf_prior/
  with_comments_clean/        docx/  pdf/  pdf_prior/
  with_comments_tracking/     docx/  pdf/  pdf_prior/
  documents.csv  comparisons.csv  pools/  notices/  PROVENANCE.json  README.md  MANIFEST.sha256.json

pdf/ holds the current Word render of each docx, pdf_prior/ the render an earlier Word build made of the same docx, under the same name. A comparison lives in the state of the compared docx.

Every file carries the id of the docx it represents, the first 10 hex digits of the sha256 of the docx bytes. A document is <id>_<name>; a Word compare of two documents is <idA>_<a>__vs__<idB>_<b>_redline_<idC>, idC being the compare's own id; the Word PDF shares its docx's stem. A tool's output for either is the Word stem plus _<tool>, and the scorer keys a candidate by stripping that suffix (pipeline.redline_key, pipeline.render_key). Names are lower-cased, folded to [a-z0-9_-] and cut at 48 characters; the original names are kept in notices/RENAMED.csv and in the names column of the tables. Byte-identical docx from several origins are one file with every origin name and set recorded on it; two different docx sharing an id fail the build.

set	what
sources_500	docx-corpus documents with their Word PDF; the stems a later Word build re-rendered keep the earlier render under pdf_prior/
en_pairs_500	the English base/next documents; the first Word pass is the reference, the second pass only fills the stems the first lacks (filled in the provenance)
redlines_a100_b10, redlines_en_500	Word compares of those documents (<base>__vs__<next>) with the Word PDF of each compared document
word_based, word_based_randomized, word_redlines_superdoc	the documents of those folders and Word's compares of their pairs (resolved through centralized_mapping.csv) with the September 2026 Word renders of the compares
word_based_0926, word_based_randomized_0926, word_redlines_superdoc_0926	the September 2026 compare run: fresh compares of the same pairs with their Word PDFs, other bytes, so other documents
nocomments, nocomments_randomized	the July 2026 Word run with comments stripped: documents and compares with their Word PDFs
fixtures_originals, fixtures_word_compares	the jubarte-first fixtures and Word's compares of them (no Word PDF exists of either); built only with --fixtures

documents.csv and comparisons.csv list every entry with its id, state, paths, sets, names, sha256 and PDF producer; comparisons.csv also carries key, base_id and next_id. pools/<set>_pairs.csv (key, base, next, docx, pdf, state) and pools/<set>_renders.csv are what bench.yaml points a run at. notices/ holds the rename record, the naming note, the ODC-By-1.0 license and the origins' own notes and logs. PROVENANCE.json records origins, counts, what was left out and why, states and the docset id of each set; the tree is pinned by MANIFEST.sha256.json. docx and PDF files are gitignored and travel with bench fixtures upload; the tables, pools, notices, provenance and manifest are tracked.

uv run bench corpus build --dry-run     # plan against the origins, copy nothing
uv run bench corpus build               # copy (a clone on APFS), write tables, pools, notices, provenance, manifest
uv run bench corpus build --fixtures /path/to/jubarte-first/_fixtures   # the fixtures sets too
uv run bench corpus check               # verify corpus/word against its manifest (exit 1 on drift)
uv run bench corpus list                # the sets with counts, docset ids and what was left out

A destination file whose bytes changed is refused unless --force; --only <set> builds a subset (a comparison set needs its sources set in the selection).

---

Scoring

1. Match candidate PDF to oracle PDF by <base>_<next> (redlines) or plain stem (docx_to_pdf, roundtrip). The denominator is the oracle document set (results/docsets.json); a document with no candidate output enters at 0 (intent-to-treat).
2. Raster each page at 144 DPI.
3. Score each page 0 to 100 as a weighted sum: SSIM 40, ink F1 20, edge IoU 15, colour 15, blob 10. A document scores 0.7 times its page mean plus 0.3 times its worst page; "Perfect (100)" counts documents within 1e-6 of 100.

Scoring core is a verbatim lift of superdoc-visual-benchmarks; tests/test_parity.py checks byte-identical behaviour. Pages present on only one side enter at 0, ink-weighted (pagefair-v2), for script_redlines, accepted_changes and roundtrip; the visual_* benchmarks rank on the raw score because cross-engine repagination is expected there. Rank ties: adjacent rows tie when the paired bootstrap interval of their median difference includes 0.

Regression gate (CI tooling, not methodology): 100 always passes; a per-document drop vs the accepted snapshot warns; an aggregate mean or median drop beyond eps = max(1e-4, 3 sigma) from results/noise_floor.json fails. Promote with uv run bench accept-scores <tool>.

Content cache

bench run reuses renders, page rasters and scored rows across runs from .bench-cache/ (next to results/, git-ignored). Entries are keyed by content, never by path: candidate sha256, oracle sha256 (and base sha256 when a base PDF joins the row), DPI, renderer id, and a scorer fingerprint (sha256 over score.py, score_v2.py, page_metrics.py, pipeline.py, raster.py, the MuPDF build and the cache schema). Editing any scoring source or upgrading PyMuPDF invalidates every score entry on its own; nothing needs clearing by hand.

A row restored from the cache carries cached: true and no raster_ns/score_ns; a restored render has cached: true and no duration_ns. Timings (render_s, raster_s, score_s, the visual_* render-speed stats) therefore come only from fresh work. Rasters are still written under the run's work dir so galleries and diagnostics read them as before.

uv run bench run --no-cache        # score everything fresh (BENCH_NO_CACHE=1 does the same)
BENCH_CACHE_DIR=/fast/disk uv run bench run
uv run bench cache                 # root, entry counts, size, scorer fingerprint
uv run bench cache --clear

bench compare never reads the cache.

Profiling the pipeline

bench profile times every stage (generate, render, raster, score) on a seeded sample of documents and prints, per run and benchmark, each stage's count, total, mean, median, p95, max and share of the run, plus the run's wall time and renderer. It is one uncached pass and never a result: nothing is appended to results/bench.jsonl, no gate runs, and recorded runs are not skipped.

uv run bench profile --run jubarte                     # 10 documents, seed 0
uv run bench profile --run jubarte --sample 25 --seed 3 --json out/profile.json
uv run bench profile --roundtrip --accept-compare      # time those stages too

The same --sample and --seed pick the same documents, so two profiles (before and after a scorer change, or CPU against a torch device) compare like for like.

Torch backend for the scorer kernels

The scorer's two heaviest kernels, the CIEDE2000 colour distance and SSIM, have a torch port behind --device on bench run, bench profile and bench compare (the gpu extra: uv sync --extra gpu). The default is the skimage path, which the parity tests lock byte for byte; the torch path reproduces it within float32 tolerance and is checked against it by tests/test_kernels.py.

uv run bench profile --run jubarte --device mps      # Apple silicon
uv run bench run --device auto                       # cuda, then mps, else the numpy path

auto is silent when no GPU is present; cpu, mps and cuda fall back to the numpy path with one warning when torch or that device is unavailable. The device is exported as BENCH_DEVICE for the command's duration only, so worker processes inherit it; the backend id (numpy, torch-mps, ...) is part of the content-cache score key, the profile report (scorer_backend) and the hardware stamp on every result. On CPU the torch kernels are no faster per core than skimage (each pool worker is limited to its share of the cores), so --device cpu is for parity checks; the gain is on a GPU.

---

Speed methodology

Data: results/speed.jsonl, results/redline_speed_bench/. Detail: docs/SPEED.md.

Mode	Measures
Warm in-process (*-inproc)	Compare work in one long-lived process
CLI	Process spawn + init + compare
WASM	In-process after load
Microbench	Small N × reps

Large-N (scripts/redline_speed_bench.ts): up to 1000 unique .docx → 5000 pairs (Mulberry32 seed 42), warmup, performance.now(), failures excluded.

node --import tsx scripts/speed-bench.ts --pairs 30 --reps 3 --out results/speed.jsonl
pnpm run redline-speed-bench:warm

---

Project map

bench.yaml                 # runs, pins, oracles
corpus/word/               # what Word produced, by document state (bench corpus build)
corpus/word_based/         # redline DOCX + LibreOffice oracle PDFs
corpus/no_comments_pdf_was_generated_by_word/  # Word-exported PDFs (docx_to_pdf)
corpus/no_comments_pdf_was_generated_by_word/renderer_corpus/  # Word/Jubarte/docxide PDFs
bench.registry.yaml        # tool identities (one id per spelling in every store)
results/bench.jsonl        # fidelity store (LFS)
results/converters.jsonl   # DOCX to PDF store (one line per report and tool)
results/docsets.json       # document set per benchmark (the ITT denominators)
results/retractions.jsonl  # runs that must never be ranked, with reasons
results/archive/           # legacy rows and their manifest (history only)
src/neurotic_docx_bench/
src/neurotic_docx_bench/ledger/         # registry, rows, policy, stats, tables, build (`bench report`)
scripts/

AGENTS.md · CONTRIBUTING.md

---

Credits

This bench exists because these projects publish their work. Every tool is measured as published, at the version pinned in bench.yaml; a score is a measurement, not an endorsement. Licenses are as each project's repository states them on 2026-10-28.

Tools measured

Project	Author	License	What the bench uses it for
jubarte-redlines	Jandira Tech (author-affiliated)	AGPL-3.0	redline generator (native CLI, WASM, lossless and AST lanes); DOCX→PDF converter
Wordcraft CLI	storytold (ArtCraft)	Apache-2.0	redline generator
docxodus	JSv4	MIT	redline generator (WASM compareDocuments, and its C# build in the speed lane)
react-docxodus-viewer	JSv4	MIT	viewer for the visual_* benchmarks
folio (@stll/folio-core, @stll/folio-react)	stella	Apache-2.0	redline generator; viewer for the visual_* benchmarks
SuperDoc (superdoc-sdk, superdoc)	Harbour Enterprises	AGPL-3.0	redline generator (Python SDK); editor for the visual_* benchmarks
superdoc-redlines	yuch85	Apache-2.0	redline generator (SuperDoc headless extract→align→apply)
docx-redline-js	AnsonLai	MIT	dropped in #88 (2026-10-28): migration-kit sibling dependency unavailable; historical results retained
redlines	Houfu Ang	MIT	text-diff redline baseline
stemma	stemma-sh	Apache-2.0	redline generator (stemma compare)
safe-docx	UseJunior	Apache-2.0	redline generator (compareDocuments)
docxide-pdf	Sverre Johann Bjørke (sverrejb)	Apache-2.0	DOCX→PDF converter; source of the second scorer's metrics (below)
rdocx	tensorbee	MIT	DOCX→PDF converter
office2pdf	Yonghye Kwon (developer0hye)	Apache-2.0	DOCX→PDF converter (Typst)
PdfItDown	Clelia (Astra) Bertelli	MIT	DOCX→PDF converter (delegates to office2pdf)
dxpdf	Nerdy Production	MIT	DOCX→PDF converter
doxx	Brandon Greenwell	MIT	listed converter; it has no PDF export, so it is never ranked
libreoffice_convert_rust	dnrops	MIT OR Apache-2.0	DOCX→PDF converter (drives LibreOffice)
PyMuPDF Pro	Artifex	commercial	DOCX→PDF converter, run unlicensed (first 3 pages only)
LibreOffice	The Document Foundation	MPL-2.0	the pinned renderer of the soffice lane and a converter

What the bench is built on

Project	Author	License	What the bench uses it for
superdoc-visual-benchmarks	SuperDoc	AGPL-3.0	the scoring core (score.py, diff.py, raster.py, reports), lifted verbatim
docxide-pdf tests/common/	sverrejb	Apache-2.0	Jaccard and text-boundary page metrics, ported to page_metrics.py and held to upstream's frozen numbers by tests/test_page_metrics.py
docx-revisions	Alvaro Fernandez Moreno	MIT	Accept All / Reject All (bench accept / reject)
nupunkt	ALEA Institute	MIT	sentence segmentation for the redlines lane
docx-corpus	SuperDoc	see the dataset card	source documents for bench try remote and the corpus
PyMuPDF	Artifex	AGPL-3.0	PDF rasterisation for both scorers
scikit-image	the scikit-image team	BSD-3-Clause	CIEDE2000 and SSIM in the scorer
Microsoft Word	Microsoft	proprietary	the oracle: every reference redline, Accept/Reject and PDF. Word is a Microsoft trademark

License

Scoring core derived from superdoc-visual-benchmarks. This repository is AGPL-3.0-only. See LICENSE.

Published scores are measurements, not endorsements. Microsoft Word is a trademark of Microsoft.