pdf2wiki 0.2.6

Convert technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown using a dual-pass MinerU pipeline plus fidelity-preserving post-processing.

pdf2wiki

Convert heavily technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown — code byte-perfect, tables intact, diagrams transcribed to Mermaid — suitable for an Obsidian vault or any Markdown-native knowledge base.

Built on MinerU with a dual-pass strategy: a pipeline pass (-m txt) takes byte-perfect code and text from the PDF's embedded text layer, a hybrid/VLM pass (--effort high) reconstructs table grids and transcribes diagrams to Mermaid, and a base-driven merge grafts the good parts together — token-verifying code so a hallucinating VLM never corrupts a listing. A six-step post-processing chain then cleans and splits the result into per-chapter files.

Why two backends instead of one: why a dual-backend pipeline.

Requirements

• Linux (native or WSL2) with an NVIDIA GPU, ≥ 8 GB VRAM, CUDA driver installed
• Python ≥ 3.11
• MinerU ≥ 3.4 on PATH
• A C compiler for vLLM's Triton JIT: sudo apt install build-essential python3-dev

Only the conversion needs the GPU — phase5, qa, and scan run anywhere Python runs. No GPU at all? offload the hybrid pass to a server (pipeline local on CPU, VLM pass remote), use a remote GPU, or — with no local setup at all — convert in the cloud via the managed mineru.net API.

Install

uv tool install pdf2wiki        # or: pip install pdf2wiki

Full steps: install pdf2wiki. The install includes every converter, cloud included — no separate extra.

Quickstart

# 1. convert one book  ->  out/my-book/my-book.md + images/ + blocks.json
pdf2wiki convert book.pdf --name my-book

# 2. preview the cleanup + chapter split (dry-run)
pdf2wiki phase5 out/my-book/my-book.md --book my-book

# 3. apply it  ->  out/my-book/chapters/00-front-matter.md, 01-….md, …
pdf2wiki phase5 out/my-book/my-book.md --book my-book --source-name book.pdf --apply

New here? Follow the guided tutorial: convert your first book.

Query the vault: the llm-wiki plugin

pdf2wiki builds a knowledge vault; the bundled llm-wiki Claude Code plugin reads one — so Claude can consult it while you plan, and review your code against it, always citing the exact [[Page-Name]]. It's the "LLM Wiki" pattern (an alternative to RAG/embeddings): plain Markdown read just-in-time, with coverage discovered from the vault itself. Standalone (works with any vault in the shape pdf2wiki produces) and MIT-licensed.

claude plugin marketplace add https://github.com/Sevthered/pdf2wiki
claude plugin install llm-wiki@pdf2wiki

Query-side (consult + review) ships now; ingest-side is a later release. Details: plugin/ · llm-wiki documentation.

Documentation

Full docs live in docs/, organized by intent (Diátaxis):

• Tutorials — convert your first book
• How-to — convert · post-process & split · batch · QA · remote GPU · offload hybrid to a server · convert in the cloud · troubleshoot
• Reference — CLI · configuration · pipeline stages · phase 5 steps · output layout
• Explanation — why dual-backend · how the merge works · design principles
• Architecture — overview with C4 diagrams
• Decisions — ADR log (MADR): why two backends, code token-verification, AGPL/MIT + subprocess, the executor port, remote hardening
• Plugin — llm-wiki (query/review a vault)

Contributing

Contributions are welcome — see CONTRIBUTING.md for setup and conventions, GOVERNANCE.md for how the project is run and who maintains it, and the Code of Conduct. Report security issues privately per SECURITY.md. Where the project is headed: the roadmap.

Status

Alpha. All stages are functional; the converter was ported from a production deployment validated on several full technical books, and the coverage gate hard-stops rather than silently dropping content. Remote (SSH) and cloud (--mineru-cloud) modes now have a verified end-to-end run on a real GPU box — a single-book conversion over SSH and a cloud dual-pass merge — but have not yet been exercised at full-book batch scale, so prefer local mode for large runs.

License

AGPL-3.0-or-later for the converter. This tool drives the MinerU CLI (AGPL-3.0) as an external process. The bundled plugin/ (the llm-wiki Claude Code plugin) is MIT-licensed — see plugin/LICENSE.