estravon-backend

Self-hosted PDF extraction backend for the Estravon Zotero plugin.

Independent project. Estravon is not affiliated with, endorsed by, or in any way connected to the Zotero project or the Corporation for Digital Scholarship. Zotero is a registered trademark of the Corporation for Digital Scholarship.

Extracts nominated sections of a book PDF to Markdown and attaches the result directly to the Zotero item — synced, versioned, always co-located with the source.

Just want to run it? Skip this page — follow the step-by-step guide at estravon.com/install instead. It covers pip install, virtual environments, and .env configuration without requiring a clone.

This README is for people who want to read the source, modify the backend, or run in editable mode.

Developer setup

git clone https://github.com/tiberavonltd/estravon-backend.git
cd estravon-backend

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

pip install -e ".[dev]"

Create a .env file in the repo root and add your API key:

MISTRAL_API_KEY=your_key_here

Get a key at console.mistral.ai (~$0.002/page).

Start the backend:

estravon --port 7766

Run the test suite:

pytest

Supported extraction backends

Only one token needs to be set. If the selected backend's token is missing, get_backend() automatically falls back to whichever of the other two is configured (fallback order: Mistral → Replicate → Datalab, Replicate → Mistral → Datalab, Datalab → Replicate → Mistral) rather than refusing to start — a warning is printed to stdout when this happens. MinerU is never part of this fallback chain — local inference is much slower than a hosted API, so it only runs when explicitly selected.

Running fully offline with MinerU (no API key)

MinerU processes the PDF entirely on your own machine — nothing is uploaded anywhere. It's not installed by default: mineru[core] pulls in a real ML stack (PyTorch, onnxruntime, ~5.5 GB installed), so it lives behind an optional extra.

You don't need a clone of this repo to use it — a plain virtual environment is enough. From any empty folder (PowerShell on Windows, or a terminal on macOS/Linux):

# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install "estravon-backend[mineru]"
estravon --port 7766 --backend mineru

# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate
pip install "estravon-backend[mineru]"
estravon --port 7766 --backend mineru

That's the whole sequence — no .env file, no API key, no repo checkout. Leave that terminal window open; it's the running server the plugin talks to on localhost:7766. First run downloads MinerU's model weights (~1 GB, cached under ~/.cache/huggingface on macOS/Linux, or C:\Users\<you>\.cache\huggingface on Windows, so later runs skip the download).

Known limitations (CPU pipeline mode):

• Slow — roughly 12 seconds per page on a modern CPU, versus seconds for a hosted API. Fine for a handful of chapters, not ideal for whole books.
• Formula rendering has a known defect: some LaTeX output has extra spacing between characters (e.g. \mathrm{s i n} instead of \mathrm{sin}), which is visibly wrong for text-mode math. Plain text and table extraction are not affected by this.
• Needs a machine with a few GB of free RAM (and, on Linux, swap configured is recommended — the pre-flight check in MinerUBackend estimates the requirement and refuses to start rather than risking an out-of-memory crash, but the estimate is a rough one, not a guarantee).

Architecture

Zotero plugin  →  POST /process  →  process_section()
                                          ↓
              MistralBackend | DatalabBackend | ReplicateBackend | MinerUBackend
                                          ↓
                                 result .md + images returned

The backend is a single-process FastHTML server, pinned to one engine per running instance. /process is synchronous — one job runs at a time and the HTTP response doesn't return until it's done or failed; a concurrent call gets 409, there is no job queue or GET /jobs/{id} polling endpoint on this server (that pattern exists on Estravon's separate hosted SaaS deployment, not in this package). GET /status exposes the current server state (idle / running / error).

Full reference, including the engine-decision table, the two-step Markdown-fetch gotcha, honest per-engine behaviour differences, and the engine-implementer contract: docs/API.md (+ machine-readable docs/openapi.json). The table below is a quick-reference summary, not the canonical source.

API reference

POST /process accepts multipart/form-data with:

⚠️ The response's files[].md_url is a path to fetch, not the Markdown text itself — see docs/API.md for the full two-step-fetch explanation.

Chunking

Sections longer than chunk_size pages are split automatically (compute_chunks() in chunking.py) into multiple labelled sub-ranges — e.g. a 200-page section at chunk_size=80 becomes three chunks, chapter_01_a.md (pages 1–80), chapter_01_b.md (81–160), chapter_01_c.md (161–200). If the whole section fits in one chunk, the output file has no suffix at all (chapter_01.md). Section names are slugified before use as filenames, so spaces and punctuation in the name you type never break file retrieval.

Content statistics

Every extraction embeds a content_stats block in the .md footer (and in state.json), computed at a tier driven by mode:

This is metadata about the extracted text, not the extraction itself — it's there so downstream tooling (search, agents, dashboards) can reason about a section without re-parsing the whole markdown file.

Large PDF handling

MistralBackend and DatalabBackend both pre-split an oversized source PDF before uploading, rather than sending the whole file and letting the API reject or silently mishandle it:

This is defense-in-depth — the Zotero plugin already trims client-side before sending anything, but the same backend is also usable standalone (scripts, other clients) where that isn't guaranteed.

Health check

curl http://localhost:7766/ping
# {"status":"ok","state":"idle","backend":"mistral"}

curl http://localhost:7766/status
# {"state":"idle","state_since_s":4.1,"backend":"mistral","last_job":{}}

Links

• Plugin repository
• End-user install guide
• estravon.com
• Report an issue

License

AGPL-3.0 — the same license as Zotero itself.