estravon-backend-benchmarks

Prerequisite: you need estravon-backend installed and reachable with the engines you want to compare configured – MinerU installed locally (pip install "estravon-backend[mineru]"), and/or MISTRAL_API_KEY / DATALAB_API_KEY / REPLICATE_API_TOKEN set as environment variables in the shell/process that calls compare() – e.g. export MISTRAL_API_KEY=... before running your script, or os.environ["MISTRAL_API_KEY"] = ... before calling compare(). This package orchestrates estravon-backend; it never extracts anything on its own, and it never reads or passes engine credentials itself – see “A note on API keys” below for exactly how they reach the engine. See estravon-backend’s docs/API.md for the engine-decision table and how to get each engine’s key.

Picking up the .env file from estravon-backend’s own setup: estravon-backend’s README has you create a .env file (e.g. MISTRAL_API_KEY=your_key_here) in that repo’s root, and estravon loads it automatically – but only from its own process’s current working directory at startup. This package spawns estravon as a subprocess from wherever your notebook’s kernel happens to be running, which usually isn’t that directory – so the reliable way to make the same key visible here is to load that .env explicitly, once, before calling compare():

from pathlib import Path
from dotenv import load_dotenv

load_dotenv(Path("~/path/to/estravon-backend/.env").expanduser())

python-dotenv ships as a dependency of estravon-backend itself, so if you’ve already installed that (this package’s own prerequisite, above), it’s already available – no separate install. .expanduser() resolves ~ portably on Linux/macOS/Windows; for a literal $HOME-style variable instead, use os.path.expandvars(...). Once loaded, the variables are in your notebook process’s own environment, and LocalEngineProcess (Mode A) inherits them when it spawns estravon – it doesn’t matter what that subprocess’s own working directory ends up being.

Install

!pip install estravon-backend-benchmarks

Also installable via git, e.g. for the latest unreleased commit on main, or for contributing (editing a notebook under nbs/):

pip install git+https://github.com/tiberavonltd/estravon-backend-benchmarks.git

To contribute, clone and install in dev mode instead:

git clone https://github.com/tiberavonltd/estravon-backend-benchmarks.git
cd estravon-backend-benchmarks
pip install -e ".[dev]"
nbdev-install-hooks   # keeps notebook diffs clean on commit

See CONTRIBUTING.md for the edit-notebook -> nbdev-export -> nbdev-test loop (nbdev 3.x only ships hyphenated console scripts – nbdev_install_hooks with an underscore will not be found).

Quickstart

compare() has two modes (see estravon-backend’s docs/API.md section “Engine selection” for why there’s no single-URL multi-engine mode – engine choice is fixed per running instance, on purpose):

• Mode A (below, the common case): pass engines=[...] and this package spawns one pinned estravon --backend <engine> --port <N> subprocess per engine for you – nothing to configure beyond having estravon-backend installed with the relevant keys/local models available.
• Mode B: pass engine_urls={"mistral": "http://host:port", ...} instead, if you already have separately-running single-engine instances.

A note on API keys – two different things share the name “api_key”:

• Engine credentials (MISTRAL_API_KEY, DATALAB_API_KEY, REPLICATE_API_TOKEN) are read by estravon-backend itself from the environment – this package never touches them. In Mode A, LocalEngineProcess spawns estravon --backend <engine> ... as a plain subprocess that inherits your calling process’s environment, so setting them before you call compare() is enough; there’s no parameter for them here because there doesn’t need to be one.
• compare()’s own api_key= parameter is a different thing entirely: it’s only used in Mode B, forwarded as the X-API-Key header to authenticate to a hosted estravon-backend instance (the same header the hosted service’s own billing/auth uses) – it has nothing to do with which engine that instance runs. In Mode A it’s hardcoded to None: locally spawned instances have no such auth layer.

The cell below is marked non-executing in this rendered copy of the notebook (no backend/engines are available in the environment that builds these docs) – otherwise it’s exactly what you’d run locally, unmodified: it uses get_artusi() (a real, public-domain 1891 cookbook PDF, downloaded and cached on first call – see estravon_bench.io) instead of a placeholder path, so there’s nothing to swap in before trying it.

from estravon_bench.compare import compare
from estravon_bench.io import get_artusi

result = compare(
    pdf_path=str(get_artusi()),   # public-domain 1891 cookbook -- downloaded/cached on first call
    page_range="1-4",
    engines=["mineru", "mistral"],   # whichever engines you have configured
    mode="balanced",
)
print(result.to_markdown_table())

This prints a side-by-side table:

| engine | time (s) | cost (usd) | local | pages | status |
|---|---|---|---|---|---|
| mineru | 34.10 | free (local) | yes | 4 | ok |
| mistral | 2.30 | $0.0080 | no | 4 | ok |

Then look at each engine’s actual output:

for r in result:
    print(f"--- {r.engine} ---")
    print(r.markdown[:500] if r.ok else f"ERROR: {r.error}")

print(result.diff("mineru", "mistral"))   # optional -- line diff for eyeballing

⚠️ The two-step fetch is handled for you – Client.fetch_markdown() already does the md_url → actual text round trip described in estravon-backend’s docs/API.md. If you’re extending client.py yourself, that’s the detail to know about; compare()’s own callers never see a bare URL.

Honest-cost labelling: local=True engines (MinerU today) show cost_usd=0.0 and the table renders “free (local)” – that means zero dollars, not zero effort or best value. A free engine that’s ten times slower is not automatically the right choice.

Scope: this is an evaluation aid for comparing engines on your own PDFs apples-to-apples – not a production layer, and not a leaderboard. It compares your PDF on your configured engines; it makes no claim about which engine is best in general.

Images

Engines that extract images (Datalab and Mistral, real-verified; any future engine automatically once it returns image_urls) have them fetched for you – result.images is a {filename: bytes} dict per engine, populated by compare() the same way result.markdown is. to_markdown_table()’s images column shows the count per engine at a glance.

No automated image comparison here – engines encode/crop/caption images differently even for “the same” figure, so this package limits itself to presence/count and letting you look:

from IPython.display import Image

row = result.get("mistral")
for filename, data in row.images.items():
    print(filename, f"{len(data)} bytes")
    display(Image(data=data))

Combining results from separate compare() calls

Running one engine at a time (different session, different machine, resource constraints) instead of passing engines=[...] all at once? Merge the results back into one ComparisonResultList afterwards – + and .merge() both return a new list without touching the originals; .extend() mutates the first list in place if you’d rather accumulate into one object as you go:

from estravon_bench.compare import compare
from estravon_bench.io import get_artusi
from estravon_bench.report import ComparisonResultList

pdf = str(get_artusi())
mineru_result  = compare(pdf, "1-4", engines=["mineru"])
mistral_result = compare(pdf, "1-4", engines=["mistral"])

combined = mineru_result + mistral_result                      # new list, both inputs untouched
# equivalent, and reads better for more than two lists:
combined = ComparisonResultList.merge(mineru_result, mistral_result)

mineru_result.extend(mistral_result)   # or: mutate mineru_result in place instead

print(combined.to_markdown_table())

Saving and loading a comparison run

ComparisonResultList.save(dir_path) writes the whole list to disk as plain files – manifest.json plus one subdirectory per engine holding result.md and an images/ folder – not a database or a binary format, so any file browser, git diff, or image viewer can inspect a saved run directly. ComparisonResultList.load(dir_path) reads it back into real ComparisonResult objects, markdown/images/metadata all restored exactly:

from estravon_bench.report import ComparisonResultList

result.save("runs/mineru_vs_mistral")

# ...later, or in a different session:
reloaded = ComparisonResultList.load("runs/mineru_vs_mistral")
print(reloaded.to_markdown_table())

Worked example: diff analysis between Mistral and Datalab on Artusi’s first 4 pages

Good test case — the Artusi title page is unusually hard (ornamental 1891 display type, blackletter-ish capitals, printer’s devices), so it stresses glyph recognition far more than body text would. Here’s what we see in the diff, grouped by kind of difference.

Structural / heading policy

Mistral promotes almost anything visually large to #: “L’ARTE DI MANGIAR BENE”, “MANUALE PRATICO PER LE FAMIGLIE” and even “PROPRIETÀ LETTERARIA” all become H1. Datalab infers a hierarchy instead — ## for the subtitle, plain paragraph for “MANUALE PRATICO…”. The stats confirm it: 8 headings vs 4. Mistral is mapping type size to heading level; Datalab is mapping document role. For Zotero/RAG chunking that matters a lot — a document where every other line is H1 gives you useless section boundaries.

Line-break fidelity

Datalab emits trailing double-spaces to preserve the couplets:

Un pasto buono ed un mezzano  
Mantengon l'uomo sano.

Mistral uses bare newlines, so those verse pairs collapse into one line when rendered. Datalab also captures the italics (*Prima digestio fit in ore.*) that Mistral drops, and escapes the decorative asterisks (Igiene \* Economia) so they don’t accidentally become emphasis.

OCR accuracy — the interesting part

they fail in different ways:

Mistral’s errors are letterform confusions (I -> A, i -> l) — classic optical failures on ornamental capitals. Datalab’s errors are language-model normalisations: “Pei” is archaic Italian printing idiom, so it “corrected” it to the more frequent “PER”; likewise “lesto”→“lieto”.

That distinction has real consequences. Optical errors look wrong and get caught. LM-normalisation errors look right and slip through. Worth flagging in a benchmark.

Note also that both engines are internally inconsistent: the same couplet appears twice in these 4 pages, and Mistral writes “Dal” on p1 but “Dai” on p3, while Datalab writes “lieto” then “lesto”. So a same-document self-consistency check would be a cheap and quite powerful quality metric to add.

Images — the biggest practical difference

Mistral: 3 images, generic alt (![img-0.jpeg](...)). Datalab: 2 images, VLM-generated descriptive alt.

Two problems on the Datalab side:

1. It dropped an image Mistral caught (the device between “IN FIRENZE” and the printer’s name) — 2 vs 3.
2. It duplicates the description into the body text, sometimes twice:

![Decorative flourish or ornament.](bench_img_002.jpg)

A decorative flourish or ornament consisting of a horizontal line with...

Decorative flourish or ornament.

That’s leaking generated English prose into an Italian document’s text stream. The contamination shows up directly in the stats — Datalab’s top keywords include lines, decorative, flourish, ornament, none of which appear in Artusi. It also inflates word_count 171→227 and fk_grade 10.3→12.6.

So the stats block is not currently comparable across engines, because one of them is measuring its own captions. If those numbers are meant to drive quality scoring, either strip alt-text before computing them, or compute them on text-only and figure-text separately.

Layout artifacts

Datalab inserts --- rules where the original has printed rules and page divisions; Mistral ignores them. Neither is wrong — it depends on whether physical layout or logical content only is wanted.

Summary judgement:

Datalab gives richer, better-structured markdown (real hierarchy, breaks, italics, figure descriptions) at the cost of speed, one missed figure, text-stream pollution, and a tendency to silently modernise archaic wording. Mistral is faster and more literal, but flattens structure and needs post-processing for headings.