paperdigest

Read any AI/ML arXiv paper without having every piece of jargon memorized. paperdigest turns a paper into a folder of plain-English, wikilinked Obsidian notes — concept-by-concept explainers, Mermaid diagrams, equations walked through line by line, and a vault-wide glossary that defines the terms as you hit them.

One generated note in Obsidian: a plain-English analogy for the concept, an auto-generated Mermaid diagram of the computation, and the paper's equation walked through line by line. Everything below is produced from the arXiv source — no hand-editing.

Built for the reader who follows AI/ML advances but isn't a specialist in every subfield: instead of bouncing between the PDF and ten browser tabs, you get the paper's ideas explained in your own vault, each unfamiliar term a click away.

See a full example

examples/9b-attention-is-all-you-need/ is a complete run on Attention Is All You Need, checked into the repo — the generated Obsidian vault (concept notes + glossary), a scaffolded project, and the run logs. It was produced entirely by a local 9B model — no cloud, no API key.

Local-first. By default paperdigest talks to a local OpenAI-compatible server (llama.cpp, Ollama, vLLM) on your own machine — no API key, no cost. Cloud backends (Anthropic, OpenAI) are opt-in for those who want them.

Quick start (local, free)

Install the CLI (Python 3.11+; pipx keeps it in an isolated environment):

pipx install arxivdigest      # or: pip install arxivdigest

(The PyPI package is named arxivdigest; the command you run is paperdigest, unchanged.)

Have a llama.cpp server running (llama-server -m model.gguf serves http://localhost:8080/v1 by default), then:

paperdigest https://arxiv.org/abs/1706.03762 --vault ~/ObsidianVault

Installing from a clone instead? python3 -m venv .venv && .venv/bin/pip install -e ., then use .venv/bin/paperdigest … for every command below.

Non-default port or engine? Point at it:

# llama.cpp on another port
.venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault --base-url http://localhost:8001/v1
# Ollama (needs the real model name)
.venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault \
  --base-url http://localhost:11434/v1 --model llama3.1

Output lands in <vault>/Papers/<year>-<title-slug>/ plus new term notes in <vault>/Glossary/. Re-running the same paper refuses to overwrite unless you pass --force. Glossary term notes are never overwritten. Pass --quiet to suppress progress output, or --max-input-chars/--max-tokens/--cache-dir to override those config keys per run (both digest and scaffold accept these). paperdigest --version prints the installed version — it is a top-level flag, not available on the subcommands.

Choosing a model & context window

Use a capable instruct model (~7B+); the pipeline needs structured JSON output, and very small models fail cleanly with an error: message. Set max_input_chars in config.toml to fit your model's context window (e.g. 24000 for ~8k-token contexts) — long papers are trimmed with a warning; set max_tokens alongside it to cap output length for the same small context. Big thinking models need the opposite: reasoning tokens count against the output cap, so if a run aborts with a truncated at max_tokens error, raise it (e.g. --max-tokens 32768 — see docs/smoke-test-crosslinks.md for a live example on a 122B).

How it works

flowchart TD
    A["arXiv paper<br>(HTML, fetched &amp; cached)"] --> B["parse into<br>sections + figures"]
    B --> C["outline (LLM):<br>concepts, jargon, TL;DR"]
    C --> D["per-concept note (LLM):<br>plain-English + equation walk-through<br>+ embedded Mermaid / ASCII diagram"]
    B --> E["figures explained<br>by a vision model"]
    C --> F["glossary (LLM):<br>define jargon not already in your vault"]
    D --> G["Obsidian vault:<br>Papers/ + Glossary/"]
    E --> G
    F --> G

The outline and glossary stages ask the server for JSON (response_format: json_object) when it's supported — that's what makes small local models reliable here. Each concept's diagram is generated as part of its note, then sanitized deterministically before it's written; figures are embedded next to the concept they illustrate.

Diagrams: Mermaid or ASCII

Structural concepts get a diagram. The default is Mermaid (renders natively in Obsidian); pass --diagram ascii (or set diagram = "ascii" in config.toml) for plain-text diagrams that can never have syntax errors. Small local models sometimes emit Mermaid with unquoted special characters in node labels, so every diagram goes through a deterministic sanitizer that auto-quotes them. Optionally, if node is on your PATH and the mermaid npm package is importable from the directory you run paperdigest in (npm install mermaid jsdom global-jsdom), each diagram is also parse-checked and a warning names any note whose diagram still fails. No node? Validation is skipped silently — the worst case is one diagram rendering as an error box in Obsidian instead of a picture.

Figures

By default, figures are downloaded from the paper and explained by a vision-capable model, with the image and its explanation embedded together in the matching concept note (or the overview, if unmatched). This needs a vision-capable backend — for local llama.cpp, serve with an --mmproj projector file. Backends without vision skip figures automatically after a warning on the first figure, and the rest of the digest completes normally. Pass --no-figures (or set figures = false in config.toml) to turn this off; it's also the budget option on cloud backends, where images are token-expensive. max_figures (default 8) caps how many figures per paper are explained, by document order.

Linking to a scaffolded project

Pass --project-dir (or set project_dir in config.toml) and notes gain a ## Build it section pointing at <project-dir>/<year>-<title-slug> — the project scaffold would create for the same paper. If that project exists on disk, concept notes also get a *Build it: …* line linking the stub module(s) whose cited §x.y paper sections match the concept, so you can jump straight from the explanation to the code that implements it.

Scaffold a research project

Turn a paper into a standalone, git-initialized project skeleton (Cookiecutter-DS-style layout, JSONL experiment tracking, stub modules mapped to paper sections):

.venv/bin/paperdigest scaffold https://arxiv.org/abs/1706.03762 --dest ~/projects

Output lands in <dest>/<year>-<title-slug>/ with its own git history. The code is a skeleton: module stubs with TODO(paper §x.y) markers plus a complete train/evaluate harness and a toy smoke experiment — not a working reimplementation. Any generation failure aborts loudly (nothing half-written); the raw model output is saved to scaffold-debug-<stage>.txt for inspection. Re-running refuses to overwrite an existing project unless you pass --force.

You don't need a big model: scaffolding is tested live end-to-end on a 9B via llama.cpp (see docs/smoke-test-scaffold.md; the cross-linking features are live-tested on a 122B via vLLM in docs/smoke-test-crosslinks.md). JSON stages request structured output (response_format: json_object) from the server when supported, which is what makes small local models reliable here; generated code is validated (syntax + prompt-echo rejection) before anything is written. Besides log_run() for JSONL experiment tracking, the scaffolded tracking.py has get_logger() for per-run log files under logs/.

A note on trust: every generated .py file is still LLM output. Before writing it, paperdigest checks it parses, rejects leftover prompt-echo lines, and scans it for dangerous calls (subprocess, socket, eval/exec, os.system, pickle.load, shutil.rmtree, and similar) plus networking/FFI imports (urllib, requests, ctypes, and similar) — aborting loudly if it finds any. That scan catches obviously hazardous patterns, not everything — read the scaffolded code before you run it.

Every project also gets a generated AGENTS.md — a coding-agent brief with a dependency-ordered implementation plan, a TODO count and cited paper sections per module, a definition of done, and ground rules for the harness. It's assembled deterministically from the same stage data, no extra LLM call. Pass --vault (or set vault in your config file) and it links back to the paper's Obsidian notes so an agent can read the plain-English explanation before touching code.

Cloud backends (optional, cost money)

Every LLM call is retried up to 2 more times with exponential backoff on transient errors (connection drops, timeouts, rate limits, 5xx); malformed JSON gets one repair round-trip; other errors fail immediately. Requests time out after 300s. On cloud backends, concept and module-stub generation runs a few requests in parallel to cut wall-clock time; against a local server it stays serial, since a single llama.cpp instance generally can't service concurrent requests.

Docker

docker build -t paperdigest .
docker run --rm -v ~/ObsidianVault:/vault \
  --add-host=host.docker.internal:host-gateway \
  paperdigest https://arxiv.org/abs/1706.03762 \
  --base-url http://host.docker.internal:8080/v1

(--add-host is needed on Linux to reach the host-side local server; for a cloud backend pass -e ANTHROPIC_API_KEY or -e OPENAI_API_KEY instead.)

Config

Copy config.example.toml to config.toml (or pass --config) so you don't retype flags. CLI flags override config values. API keys are env-only and never read from config files.

Inputs

v1 accepts arXiv URLs (abs/pdf/html) or bare IDs (1706.03762). It uses the paper's arXiv HTML rendering (ar5iv as fallback); papers with no HTML rendering fail with a clear message. Fetched papers are cached in ~/.cache/paperdigest/ (configurable via cache_dir in config.toml); pass --refresh to re-fetch and overwrite the cached HTML, e.g. after arXiv publishes a revision.

Development

.venv/bin/pip install -e ".[dev]"
.venv/bin/pytest          # fully offline: fixtures + fake LLM backend

Known minor cleanups (all reviewed, none load-bearing) are tracked in TODOS.md.