pdf2wiki 0.1.0

Convert technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown using a dual-pass MinerU pipeline plus fidelity-preserving post-processing.

pdf2wiki

Convert heavily technical books (native-text PDFs) into clean, chapter-split, LLM-ready Markdown — code byte-perfect, tables intact, diagrams transcribed to Mermaid — suitable for an Obsidian vault or any Markdown-native knowledge base.

Built on MinerU with a dual-pass strategy: a pipeline pass (-m txt) takes byte-perfect code and text from the PDF's embedded text layer, a hybrid/VLM pass (--effort high) reconstructs table grids and transcribes diagrams to Mermaid, and a base-driven merge grafts the good parts together — token-verifying code so a hallucinating VLM never corrupts a listing. A six-step post-processing chain then cleans and splits the result into per-chapter files.

Why two backends instead of one: why a dual-backend pipeline.

Requirements

• Linux (native or WSL2) with an NVIDIA GPU, ≥ 8 GB VRAM, CUDA driver installed
• Python ≥ 3.11
• MinerU ≥ 3.4 on PATH
• A C compiler for vLLM's Triton JIT: sudo apt install build-essential python3-dev

Only the conversion needs the GPU — phase5, qa, and scan run anywhere Python runs.

Install

uv tool install pdf2wiki        # or: pip install pdf2wiki

Full steps: install pdf2wiki.

Quickstart

# 1. convert one book  ->  out/my-book/my-book.md + images/ + blocks.json
pdf2wiki convert book.pdf --name my-book

# 2. preview the cleanup + chapter split (dry-run)
pdf2wiki phase5 out/my-book/my-book.md --book my-book

# 3. apply it  ->  out/my-book/chapters/00-front-matter.md, 01-….md, …
pdf2wiki phase5 out/my-book/my-book.md --book my-book --source-name book.pdf --apply

New here? Follow the guided tutorial: convert your first book.

Documentation

Full docs live in docs/, organized by intent (Diátaxis):

• Tutorials — convert your first book
• How-to — convert · post-process & split · batch · QA · remote GPU · troubleshoot
• Reference — CLI · configuration · pipeline stages · phase 5 steps · output layout
• Explanation — why dual-backend · how the merge works · design principles
• Architecture — overview with C4 diagrams

Status

Alpha. All stages are functional; the converter was ported from a production deployment validated on several full technical books, and the coverage gate hard-stops rather than silently dropping content. Remote mode is experimental — no full public end-to-end run yet; prefer local mode.

License

AGPL-3.0-or-later. This tool drives the MinerU CLI (AGPL-3.0) as an external process.