pdf2wiki

Convert heavily technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown. Code stays byte-perfect, tables stay intact, and diagrams are transcribed to Mermaid. The output suits an Obsidian vault or any Markdown-native knowledge base.

Built on MinerU with a dual-pass strategy. A pipeline pass (-m txt) takes byte-perfect code and text from the PDF's embedded text layer. A hybrid/VLM pass (--effort high) reconstructs table grids and transcribes diagrams to Mermaid. A base-driven merge then grafts the good parts together, and token-verifies the code so that a VLM which hallucinates never corrupts a listing. A six-step post-processing chain cleans the result and splits it into per-chapter files.

Why two backends instead of one: why a dual-backend pipeline.

Requirements

• Linux (native or WSL2) with an NVIDIA GPU, ≥ 8 GB VRAM, CUDA driver installed
• Python ≥ 3.11
• MinerU ≥ 3.4 on PATH
• A C compiler for vLLM's Triton JIT: sudo apt install build-essential python3-dev

Only the conversion needs the GPU — phase5, qa, and scan run anywhere Python runs. No GPU at all? offload the hybrid pass to a server, which keeps the pipeline pass local on CPU. Or use a remote GPU. Or, with no local setup at all, convert in the cloud via the managed mineru.net API.

Install

uv tool install pdf2wiki        # or: pip install pdf2wiki

Full steps: install pdf2wiki. The install includes every converter, cloud included — no separate extra.

Quickstart

# 1. convert one book  ->  out/my-book/my-book.md + images/ + blocks.json
pdf2wiki convert book.pdf --name my-book

# 2. preview the cleanup + chapter split (dry-run)
pdf2wiki phase5 out/my-book/my-book.md --book my-book

# 3. apply it  ->  out/my-book/chapters/00-front-matter.md, 01-….md, …
pdf2wiki phase5 out/my-book/my-book.md --book my-book --source-name book.pdf --apply

New here? Follow the guided tutorial: convert your first book.

Query the vault: the llm-wiki plugin

pdf2wiki builds a knowledge vault. The bundled llm-wiki Claude Code plugin reads one. Claude can consult it while you plan, and review your code against it. It always cites the exact [[Page-Name]]. It's the "LLM Wiki" pattern (an alternative to RAG/embeddings): plain Markdown read just-in-time, with coverage discovered from the vault itself. Standalone (works with any vault in the shape pdf2wiki produces) and MIT-licensed.

claude plugin marketplace add https://github.com/Sevthered/pdf2wiki
claude plugin install llm-wiki@pdf2wiki

The query side (consult + review) ships now. The ingest side is a later release. Details: plugin/ · llm-wiki documentation.

Documentation

Full docs live in docs/, organized by intent (Diátaxis):

• Tutorials — convert your first book
• How-to — convert · post-process & split · batch · QA · remote GPU · offload hybrid to a server · convert in the cloud · troubleshoot
• Reference — CLI · configuration · pipeline stages · phase 5 steps · output layout
• Explanation — why dual-backend · how the merge works · design principles
• Architecture — overview with C4 diagrams
• Decisions — ADR log (MADR): why two backends, code token-verification, AGPL/MIT + subprocess, the executor port, remote hardening
• Plugin — llm-wiki (query/review a vault)

Contributing

Contributions are welcome — see CONTRIBUTING.md for setup and conventions, GOVERNANCE.md for how the project is run and who maintains it, and the Code of Conduct. Where the project is headed: the roadmap.

Reporting bugs. Found a defect or have a feature request? Open a GitHub issue — the bug-report template prompts for the version, environment, steps to reproduce, and expected vs actual behavior. Security issues must be reported privately per SECURITY.md, never as a public issue.

Repository scope. pdf2wiki is a single repository with no separate subprojects or additional repositories. The bundled llm-wiki Claude Code plugin ships in-repo under plugin/.

Status

Alpha. All stages are functional. The converter was ported from a production deployment validated on several full technical books. The coverage gate hard-stops on a gap rather than a silent loss.

Remote (SSH) and cloud (--mineru-cloud) modes each have a verified end-to-end run on a real GPU box. Those runs were a single-book conversion over SSH, and a cloud dual-pass merge. Neither mode has been exercised at full-book batch scale yet, so prefer local mode for large runs.

License

AGPL-3.0-or-later for the converter. This tool drives the MinerU CLI (AGPL-3.0) as an external process. The bundled plugin/ (the llm-wiki Claude Code plugin) is MIT-licensed. See plugin/LICENSE.