📚 DABOOK

Compile any book-like PDF into a traceable, validated, editable Book Graph, then export it as ML-ready datasets (raw / pretrain / sft / rag / code) — resumable, multi-book, parallel, with a live localhost control room.

Version

Current release: v0.1.0 (Pre-Alpha)

This is early-stage software. The core pipeline (S00–S13) and worker/supervisor runtime are functional. Some features listed in the CLI help (e.g. cache gc, --watch, --autotune) are planned but not yet implemented.

See CHANGELOG.md for release history.

⚡ Quick Start

Prerequisites

• Python 3.11 or newer
• SQLite 3.35 or newer (ships with Python ≥ 3.12; check with python -c "import sqlite3; print(sqlite3.sqlite_version)")
• ~5 GB free disk space per book

Install

# Recommended: install with uv
uv tool install dabook

# Or with pip
pip install dabook

Run

# Process a single book — starts workers, launches dashboard
dabook run my_book.pdf

# Process a whole directory
dabook run ./books/

# Resume an interrupted run with zero re-work
dabook resume

# Check that your environment is ready
dabook doctor

The live control room dashboard opens automatically at http://127.0.0.1:8765.

🌟 Key Features

• Guaranteed Resumable — kill -9 or power loss at any instant; dabook resume picks up with zero re-work thanks to two-phase atomic filesystem commits and content-addressed caching.
• Zero-Redo Caching — Same PDF + same config = instant skip. The cache key is sha256(pdf) + sha256(stage_params).
• Multi-Book Parallelism — Process batches of books concurrently. Worker counts, book concurrency, and memory ceilings are all live-adjustable from the dashboard without restart.
• Live Localhost Control Room — Real-time SSE dashboard at http://127.0.0.1:8765 showing task throughput, worker pool status, per-book progress, error logs, and ETA.
• Full Provenance Chain — Every token traces back: PDF → Page → BBox → Block → Node → Export Record.
• Multiple ML Export Formats — Produces five JSONL files per book (see Export Formats below).
• Human-Editable — The Book Graph is plain JSON; you can inspect and patch any node and re-compile.

🏗️ Pipeline Stages (S00–S13)

PDF → S00 → S01 → S02 (×shards) → S03 → S04 → S05 → S06
                                                        ↓
          datasets ← S13 ← S12 ← S11 ← S10 ← S09 ← S08 ← S07

All stages run through the same crash-safe commit protocol: write to .partial/ → atomic rename → write _SUCCESS. A stage output is valid if and only if _SUCCESS exists.

📦 Export Formats

S13 produces five JSONL files inside <workspace>/books/<sha12>/datasets/:

A manifest.json and <workspace>/datasets/manifest.json (consolidated across all books) are also written.

💻 CLI Reference

# ── Processing ──────────────────────────────────────────────────────
dabook run <path> [<path>...]          # Process PDFs or directories
  --workspace / -w <dir>              # Workspace dir (default: ./dabook_workspace)
  --profile fast|balanced|accurate|lowram
  --workers-cpu N                     # Override CPU worker count
  --workers-gpu N                     # Override GPU worker count
  --book-concurrency N                # Max books processed at once
  --port 8765                         # Dashboard port
  --no-ui                             # Skip dashboard
  --no-open                           # Don't auto-open browser

dabook resume [-w <dir>]              # Resume all work in a workspace

# ── Monitoring ──────────────────────────────────────────────────────
dabook status [-w <dir>] [--json]     # Print book/task counts by state
dabook ui [-w <dir>] [--port 8765]    # Start dashboard for an existing workspace
dabook doctor [-w <dir>]              # Check env, deps, SQLite version, disk space

# ── Queue Control ───────────────────────────────────────────────────
dabook pause [-w <dir>]               # Pause the processing queue
dabook continue [-w <dir>]            # Resume a paused queue
dabook stop [-w <dir>]                # Soft-stop (finish current task then exit)

# ── Recovery ────────────────────────────────────────────────────────
dabook retry [-w <dir>] [--dead] [--failed] [--book <sha_prefix>]

# ── Cache ────────────────────────────────────────────────────────────
dabook cache stats [-w <dir>]         # Count committed stage artifacts
dabook cache verify [-w <dir>]        # Deep-verify SHA-256 of all artifacts
dabook cache gc                       # (planned, not yet implemented)

⚙️ Worker Pools

Workers are spread across four resource classes. The supervisor spawns and drains them automatically; all counts are live-adjustable in the dashboard.

The supervisor uses a crash loop breaker: if a worker lane crashes 3+ times within 5 minutes, spawning is paused for 30 seconds before retrying.

🎯 Design Principles

1. The block is the unit of meaning, never the page. Page boundaries are printing artifacts; semantic blocks transcend them.
2. Deterministic first, AI second. Regex, font heuristics, and bbox geometry run before any neural model.
3. Strict source-faithfulness. Raw text is never discarded; all generated/inferred content is flagged separately.
4. Three text representations. raw (exact PDF glyphs) → normalized (ligatures, hyphens fixed) → clean (furniture stripped, Unicode NFC).
5. Infrastructure before intelligence. WAL-mode SQLite, process leases, atomic commits, and crash recovery come before ML pipelines.

🗄️ State Store

All state lives in <workspace>/state.db (SQLite, WAL mode). Key tables:

🤝 Contributing

git clone https://github.com/brovk2008/Dabook.git
cd Dabook
uv venv --python 3.11
source .venv/bin/activate        # Windows: .\.venv\Scripts\Activate.ps1
uv pip install -e ".[dev]"
pytest                           # run the test suite
pytest -m "not slow"             # skip crash-safety tests

See CONTRIBUTING.md, CODE_OF_CONDUCT.md, SECURITY.md, and SUPPORT.md.

⚖️ License

Distributed under the Apache License 2.0. See LICENSE for details. Optional third-party plugins or proprietary model weights have separate licenses.