pdf2wiki 0.2.0

Convert technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown using a dual-pass MinerU pipeline plus fidelity-preserving post-processing.

pdf2wiki

Convert heavily technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown — code byte-perfect, tables intact, diagrams transcribed to Mermaid — suitable for an Obsidian vault or any Markdown-native knowledge base.

Built on MinerU with a dual-pass strategy: a pipeline pass (-m txt) takes byte-perfect code and text from the PDF's embedded text layer, a hybrid/VLM pass (--effort high) reconstructs table grids and transcribes diagrams to Mermaid, and a base-driven merge grafts the good parts together — token-verifying code so a hallucinating VLM never corrupts a listing. A six-step post-processing chain then cleans and splits the result into per-chapter files.

Why two backends instead of one: why a dual-backend pipeline.

Requirements

• Linux (native or WSL2) with an NVIDIA GPU, ≥ 8 GB VRAM, CUDA driver installed
• Python ≥ 3.11
• MinerU ≥ 3.4 on PATH
• A C compiler for vLLM's Triton JIT: sudo apt install build-essential python3-dev

Only the conversion needs the GPU — phase5, qa, and scan run anywhere Python runs. No GPU at all? offload the hybrid pass to a server (pipeline local on CPU, VLM pass remote), use a remote GPU, or — with no local setup at all — convert in the cloud via the managed mineru.net API.

Install

uv tool install pdf2wiki        # or: pip install pdf2wiki

Full steps: install pdf2wiki.

Quickstart

# 1. convert one book  ->  out/my-book/my-book.md + images/ + blocks.json
pdf2wiki convert book.pdf --name my-book

# 2. preview the cleanup + chapter split (dry-run)
pdf2wiki phase5 out/my-book/my-book.md --book my-book

# 3. apply it  ->  out/my-book/chapters/00-front-matter.md, 01-….md, …
pdf2wiki phase5 out/my-book/my-book.md --book my-book --source-name book.pdf --apply

New here? Follow the guided tutorial: convert your first book.

Query the vault: the llm-wiki plugin

pdf2wiki builds a knowledge vault; the bundled llm-wiki Claude Code plugin reads one — so Claude can consult it while you plan, and review your code against it, always citing the exact [[Page-Name]]. It's the "LLM Wiki" pattern (an alternative to RAG/embeddings): plain Markdown read just-in-time, with coverage discovered from the vault itself. Standalone (works with any vault in the shape pdf2wiki produces) and MIT-licensed.

claude plugin marketplace add https://github.com/Sevthered/pdf2wiki
claude plugin install llm-wiki@pdf2wiki

Query-side (consult + review) ships now; ingest-side is a later release. Details: plugin/ · llm-wiki documentation.

Documentation

Full docs live in docs/, organized by intent (Diátaxis):

• Tutorials — convert your first book
• How-to — convert · post-process & split · batch · QA · remote GPU · offload hybrid to a server · convert in the cloud · troubleshoot
• Reference — CLI · configuration · pipeline stages · phase 5 steps · output layout
• Explanation — why dual-backend · how the merge works · design principles
• Architecture — overview with C4 diagrams
• Plugin — llm-wiki (query/review a vault)

Status

Alpha. All stages are functional; the converter was ported from a production deployment validated on several full technical books, and the coverage gate hard-stops rather than silently dropping content. Remote mode is experimental — no full public end-to-end run yet; prefer local mode.

License

AGPL-3.0-or-later for the converter. This tool drives the MinerU CLI (AGPL-3.0) as an external process. The bundled plugin/ (the llm-wiki Claude Code plugin) is MIT-licensed — see plugin/LICENSE.