estravon-backend-benchmarks

Prerequisite: you need estravon-backend installed and reachable with the engines you want to compare configured – MinerU installed locally (pip install "estravon-backend[mineru]"), and/or MISTRAL_API_KEY / DATALAB_API_KEY / REPLICATE_API_TOKEN set as environment variables in the shell/process that calls compare() – e.g. export MISTRAL_API_KEY=... before running your script, or os.environ["MISTRAL_API_KEY"] = ... before calling compare(). This package orchestrates estravon-backend; it never extracts anything on its own, and it never reads or passes engine credentials itself – see “A note on API keys” below for exactly how they reach the engine. See estravon-backend’s docs/API.md for the engine-decision table and how to get each engine’s key.

Picking up the .env file from estravon-backend’s own setup: estravon-backend’s README has you create a .env file (e.g. MISTRAL_API_KEY=your_key_here) in that repo’s root, and estravon loads it automatically – but only from its own process’s current working directory at startup. This package spawns estravon as a subprocess from wherever your notebook’s kernel happens to be running, which usually isn’t that directory – so the reliable way to make the same key visible here is to load that .env explicitly, once, before calling compare():

from pathlib import Path
from dotenv import load_dotenv

load_dotenv(Path("~/path/to/estravon-backend/.env").expanduser())

python-dotenv ships as a dependency of estravon-backend itself, so if you’ve already installed that (this package’s own prerequisite, above), it’s already available – no separate install. .expanduser() resolves ~ portably on Linux/macOS/Windows; for a literal $HOME-style variable instead, use os.path.expandvars(...). Once loaded, the variables are in your notebook process’s own environment, and LocalEngineProcess (Mode A) inherits them when it spawns estravon – it doesn’t matter what that subprocess’s own working directory ends up being.

Install

!pip install estravon-backend-benchmarks

Also installable via git, e.g. for the latest unreleased commit on main, or for contributing (editing a notebook under nbs/):

pip install git+https://github.com/tiberavonltd/estravon-backend-benchmarks.git

To contribute, clone and install in dev mode instead:

git clone https://github.com/tiberavonltd/estravon-backend-benchmarks.git
cd estravon-backend-benchmarks
pip install -e ".[dev]"
nbdev-install-hooks   # keeps notebook diffs clean on commit

See CONTRIBUTING.md for the edit-notebook -> nbdev-export -> nbdev-test loop (nbdev 3.x only ships hyphenated console scripts – nbdev_install_hooks with an underscore will not be found).

Quickstart

compare() has two modes (see estravon-backend’s docs/API.md section “Engine selection” for why there’s no single-URL multi-engine mode – engine choice is fixed per running instance, on purpose):

• Mode A (below, the common case): pass engines=[...] and this package spawns one pinned estravon --backend <engine> --port <N> subprocess per engine for you – nothing to configure beyond having estravon-backend installed with the relevant keys/local models available.
• Mode B: pass engine_urls={"mistral": "http://host:port", ...} instead, if you already have separately-running single-engine instances.

A note on API keys – two different things share the name “api_key”:

• Engine credentials (MISTRAL_API_KEY, DATALAB_API_KEY, REPLICATE_API_TOKEN) are read by estravon-backend itself from the environment – this package never touches them. In Mode A, LocalEngineProcess spawns estravon --backend <engine> ... as a plain subprocess that inherits your calling process’s environment, so setting them before you call compare() is enough; there’s no parameter for them here because there doesn’t need to be one.
• compare()’s own api_key= parameter is a different thing entirely: it’s only used in Mode B, forwarded as the X-API-Key header to authenticate to a hosted estravon-backend instance (the same header the hosted service’s own billing/auth uses) – it has nothing to do with which engine that instance runs. In Mode A it’s hardcoded to None: locally spawned instances have no such auth layer.

The cell below is marked non-executing in this rendered copy of the notebook (no backend/engines are available in the environment that builds these docs) – otherwise it’s exactly what you’d run locally, unmodified: it uses get_artusi() (a real, public-domain 1891 cookbook PDF, downloaded and cached on first call – see estravon_bench.io) instead of a placeholder path, so there’s nothing to swap in before trying it.

from estravon_bench.compare import compare
from estravon_bench.io import get_artusi

result = compare(
    pdf_path=str(get_artusi()),   # public-domain 1891 cookbook -- downloaded/cached on first call
    page_range="1-4",
    engines=["mineru", "mistral"],   # whichever engines you have configured
    mode="balanced",
)
print(result.to_markdown_table())

This prints a side-by-side table:

| engine | time (s) | cost (usd) | local | pages | status |
|---|---|---|---|---|---|
| mineru | 34.10 | free (local) | yes | 4 | ok |
| mistral | 2.30 | $0.0080 | no | 4 | ok |

Then look at each engine’s actual output:

for r in result:
    print(f"--- {r.engine} ---")
    print(r.markdown[:500] if r.ok else f"ERROR: {r.error}")

print(result.diff("mineru", "mistral"))   # optional -- line diff for eyeballing

⚠️ The two-step fetch is handled for you – Client.fetch_markdown() already does the md_url → actual text round trip described in estravon-backend’s docs/API.md. If you’re extending client.py yourself, that’s the detail to know about; compare()’s own callers never see a bare URL.

Honest-cost labelling: local=True engines (MinerU today) show cost_usd=0.0 and the table renders “free (local)” – that means zero dollars, not zero effort or best value. A free engine that’s ten times slower is not automatically the right choice.

Scope: this is an evaluation aid for comparing engines on your own PDFs apples-to-apples – not a production layer, and not a leaderboard. It compares your PDF on your configured engines; it makes no claim about which engine is best in general.

Images

Engines that extract images (Datalab and Mistral, real-verified; any future engine automatically once it returns image_urls) have them fetched for you – result.images is a {filename: bytes} dict per engine, populated by compare() the same way result.markdown is. to_markdown_table()’s images column shows the count per engine at a glance.

No automated image comparison here – engines encode/crop/caption images differently even for “the same” figure, so this package limits itself to presence/count and letting you look:

from IPython.display import Image

row = result.get("mistral")
for filename, data in row.images.items():
    print(filename, f"{len(data)} bytes")
    display(Image(data=data))

Combining results from separate compare() calls

Running one engine at a time (different session, different machine, resource constraints) instead of passing engines=[...] all at once? Merge the results back into one ComparisonResultList afterwards – + and .merge() both return a new list without touching the originals; .extend() mutates the first list in place if you’d rather accumulate into one object as you go:

from estravon_bench.compare import compare
from estravon_bench.io import get_artusi
from estravon_bench.report import ComparisonResultList

pdf = str(get_artusi())
mineru_result  = compare(pdf, "1-4", engines=["mineru"])
mistral_result = compare(pdf, "1-4", engines=["mistral"])

combined = mineru_result + mistral_result                      # new list, both inputs untouched
# equivalent, and reads better for more than two lists:
combined = ComparisonResultList.merge(mineru_result, mistral_result)

mineru_result.extend(mistral_result)   # or: mutate mineru_result in place instead

print(combined.to_markdown_table())

Saving and loading a comparison run

ComparisonResultList.save(dir_path) writes the whole list to disk as plain files – manifest.json plus one subdirectory per engine holding result.md and an images/ folder – not a database or a binary format, so any file browser, git diff, or image viewer can inspect a saved run directly. ComparisonResultList.load(dir_path) reads it back into real ComparisonResult objects, markdown/images/metadata all restored exactly:

from estravon_bench.report import ComparisonResultList

result.save("runs/mineru_vs_mistral")

# ...later, or in a different session:
reloaded = ComparisonResultList.load("runs/mineru_vs_mistral")
print(reloaded.to_markdown_table())

Worked examples

Real, live-run engine comparisons – actual findings, not illustrative examples – live in a separate notebook: 06_worked_examples.ipynb.