distl

Compile a PDF or DOCX document into individually addressable, versioned Knowledge Objects — so a coding agent can fetch one fact instead of reading the whole file.

Read this before adopting it: this README documents what is actually implemented and measured today — not a roadmap, not a pitch. The Not built yet section is as important as the feature list above it.

Project status

First release v0.1.0rc1 (pre-release) published to PyPI as distlk; single maintainer, actively developed. The core pipeline (ingest → chunk → extract → validate → write, plus incremental update) is built and tested; what's not built yet is listed explicitly below, not left for you to discover the hard way.

214 tests, 94% statement coverage (pytest --cov=src/distl) — the lowest-covered module is cli.py at 54%, since most of its logic is thin argument wiring already exercised indirectly through the pipeline tests, not untested business logic. CI (.github/workflows/ci.yml) runs lint + the full test suite with coverage on every push/PR to master; pushing a v* tag runs .github/workflows/release.yml, which gates on the same checks, builds the sdist + wheel, publishes to PyPI as distlk, and files a GitHub Release.

---

Table of contents

• Project status
• The problem
• What distl does
• Quick start
• How it works
• CLI reference
• Configuration
• Output layout
• Design decisions, briefly
• Measured, not claimed
• Not built yet
• Development
• Contributing
• License

---

The problem

Coding agents are usually handed one of two bad options when a task depends on a spec, an RFC, or a requirements doc: dump the entire PDF into context (expensive, unfocused, drowns the one relevant paragraph in fifty irrelevant ones), or rely on the agent to grep/skim it (unreliable, easy to miss a page-12 constraint that contradicts a page-40 assumption).

distl exists to produce a third option: compile the document once into small, individually addressable, versioned units — a requirement, a business rule, a workflow step — each one citing exactly where it came from, so an agent (or a human) can pull the three relevant objects instead of the whole 140-page PDF.

What distl does

• Ingests PDF/DOCX via Docling, preserving reading order, heading hierarchy, page/section anchors, and table content (rendered as Markdown). Pictures are detected and their image bytes preserved on disk regardless.
• Reads pictures (opt-in): with DISTL_VISION_ENABLED=true, a picture's content (a chart, diagram, or screenshot) is read by a vision-capable LLM (cheap tier first, escalating to a stronger model only on low self-reported confidence) and turned into real Knowledge Object(s) — the same structural validation and id-dedup guarantees every text-derived object gets, tagged extraction_method: vision-picture-extract-v1. Default off; a picture that isn't read (or fails to be) still produces today's visible gap, image preserved on disk regardless.
• Chunks at heading boundaries under a configurable token budget — a chunk never silently merges two unrelated sections into one citation.
• Extracts Knowledge Objects per chunk via an LLM router (Gemini primary, OpenRouter fallback) with retry/backoff, automatic provider fallback, and a hard per-run cost ceiling that halts cleanly and resumes later without re-processing anything already written.
• Validates structurally before anything is written — a malformed object becomes a visible gap in validation/gaps.yaml, never a silent drop and never a file on disk.
• Updates incrementally: distl update re-diffs a changed source, re-extracts only the sections that actually changed, bumps versions only on real content changes, fuzzy-matches objects across revisions, and routes anything conflicting with a human edit to a merge-conflict file instead of overwriting it.
• Governs tags through a reviewed, curated vocabulary instead of letting the LLM invent an ever-growing, inconsistent tag soup.
• Shards large manifests automatically once a package crosses a token budget, so the index stays cheap to read even as a package grows.

A concrete example

(Illustrative — the schema and pipeline behavior below are real; this specific document is a constructed example, not a real customer document.)

Given a source paragraph like:

2.3 Password Reset. A user who has forgotten their password may request a reset link via the "Forgot password" control on the sign-in screen. The link expires after 30 minutes. After a successful reset, all other active sessions for that account are terminated.

distl compile produces requirements/REQ-7.md:

---
id: REQ-7
title: Password reset link expires after 30 minutes
status: active
description: >
  A user who forgot their password can request a reset link from the "Forgot password"
  control on the sign-in screen. The link expires 30 minutes after issuance. On a
  successful reset, all other active sessions for the account are terminated.
confidence: 0.93
source_location:
  document: requirements.pdf
  page: 4
  section: "2.3 Password Reset"
extraction_method: chunk-summarize-v1
version: 1
extraction_provider: google-ai-studio
extraction_model: gemini-3.5-flash
tags: [authentication, session-management]
---

An agent asking "what happens to other sessions on password reset?" reads one ~15-line file with an exact page citation — not a 140-page PDF.

Quick start

Requires Python 3.12+.

pip install distlk                        # published release
pip install -e ".[dev]"                   # or: development install from this repo

cp .env.example .env   # then fill in DISTL_GEMINI_API_KEY at minimum
distl compile requirements.pdf --output ./knowledge-package

Sample output (illustrative — object ids/titles/timings vary per document and run; the line format itself is real, from cli.py's progress reporter):

Parsing requirements.pdf...
[1/6] extracted REQ-1 'User accounts require email verification' (confidence 0.95, page 1) via google-ai-studio, 2.1s
[2/6] extracted REQ-2 'Password minimum length is 12 characters' (confidence 0.91, page 2) via google-ai-studio, 1.8s
...
Compiled 6 knowledge object(s) to knowledge-package

A more realistic workflow

Compiling once is the easy case. What actually happens across a document's lifetime (illustrative counts/comments below — the commands and CLI output format are real, the specific numbers are made up for the example):

# Initial compile hits the cost ceiling partway through a large document
distl compile spec-v1.pdf --output ./kp
# Halted: fallback cost ceiling exceeded -- see validation/cost-ceiling-alert.md

# Resume later without re-processing what's already written
distl compile ./kp --resume

# The source doc gets revised -- re-extract only what changed
distl update ./kp spec-v2.pdf
# 3 updated, 1 new, 1 deprecated, 0 merge conflict(s)

# A reviewer had hand-edited one object; the update flagged a real conflict
distl merge-conflicts ./kp

# New tags the LLM proposed need a human decision before they're trusted
distl tags review ./kp

# Spot-check a sample for citation accuracy before trusting the package
distl audit ./kp --sample 0.2

More, including a script for the sequence above and a programmatic (non-CLI) query example: see examples/.

How it works

flowchart LR
    A["PDF / DOCX"] --> B["Ingest<br/>(Docling)"]
    B --> C["Chunk<br/>(heading-boundary)"]
    C --> D["Extract<br/>(LLM router)"]
    D -->|"rate limit / timeout / malformed"| E["Fallback provider"]
    D --> F["Validate<br/>(structural)"]
    E --> F
    F -->|fail| G["validation/gaps.yaml"]
    F -->|pass| H["Knowledge Object .md"]
    H --> I["manifest.yaml"]

Two guardrails run the whole time, not shown above for clarity: a cost ceiling that halts the run cleanly (resumable) if it would be exceeded, and a validation gate that means nothing malformed ever reaches disk.

Full module map (what's real vs. an empty placeholder) and the design rationale behind the trickier decisions: docs/ARCHITECTURE.md.

CLI reference

Command	What it does
distl compile <file> --output <dir>	Compile a source document into a new Knowledge Package.
distl compile <dir> --resume [--source <path>]	Resume a run halted by the cost ceiling; --source re-points at a moved source document.
distl update <dir> <file>	Re-compile only the sections of <dir> that changed in <file>.
distl audit <dir> --sample <fraction>	Interactively sample objects and judge traceability to source; reports a hallucination rate.
distl merge-conflicts <dir>	List objects where a human edit and a new extraction disagreed.
distl tags review <dir>	Accept or reject LLM-proposed tags not yet in the controlled vocabulary.
distl baseline run <file> [-c N ...]	Run at one or more concurrency levels, recording cost/latency/token metrics.
distl baseline report --runs-dir <dir>	Aggregate baseline runs into a Markdown table.

Configuration

Environment variables, prefixed DISTL_, loaded via Pydantic Settings (src/distl/config.py):

Variable	Default	Meaning
DISTL_GEMINI_API_KEY	(none)	Primary extraction provider.
DISTL_OPENROUTER_API_KEY	(none)	Fallback provider. Unset = fallback skipped, recorded as a gap, never an error.
DISTL_GEMINI_MODEL	gemini-3.5-flash	Primary model.
DISTL_OPENROUTER_FALLBACK_MODEL	openai/gpt-5-mini	Fallback model.
DISTL_CHUNK_SIZE_TOKENS	2000	Token budget per chunk before a heading-group splits further.
DISTL_MAX_CONCURRENCY	5	Concurrent chunk extractions per run.
DISTL_FALLBACK_COST_CEILING_USD	5.00	Hard per-run spend ceiling; halts cleanly (resumable) if it would be exceeded.
DISTL_RATE_LIMIT_RETRY_ATTEMPTS	2	Retries on a rate-limit error before falling back.
DISTL_TIMEOUT_RETRY_ATTEMPTS	1	Retries on a timeout before falling back.
DISTL_MANIFEST_TOKEN_BUDGET	8000	Manifest size threshold before it shards into manifest.d/.
DISTL_ID_MATCH_SIMILARITY_THRESHOLD	0.80	Fuzzy title-similarity cutoff for distl update's object matcher. Not yet tuned against real collision data.
DISTL_CONTROLLED_TAG_VOCABULARY_PATH	./tags.yaml	Relative paths resolve against the package directory, not the current working directory.
DISTL_VISION_ENABLED	false	Opt-in: read picture content via a vision LLM instead of just preserving the image.
DISTL_VISION_MODEL_TIER1	qwen/qwen3-vl-8b-instruct	Cheap/fast vision model, tried first.
DISTL_VISION_MODEL_TIER2	qwen/qwen3-vl-235b-a22b-instruct	Escalation target when tier 1's self-reported confidence is below the threshold.
DISTL_VISION_CONFIDENCE_THRESHOLD	0.70	Below this, tier 1's result is discarded and tier 2 is tried instead.
DISTL_VISION_ESTIMATED_COST_TIER1_USD	0.01	Pre-call cost-ceiling check estimate for tier 1 (real cost is reconciled after the call).
DISTL_VISION_ESTIMATED_COST_TIER2_USD	0.05	Same, for tier 2.
DISTL_VISION_RETRY_ATTEMPTS	2	Mirrors DISTL_RATE_LIMIT_RETRY_ATTEMPTS: retries a vision call on transient rates/timeout/malformed-response failures (default 2 = one retry).

Output layout

knowledge-package/
├── manifest.yaml                 # index: objects, tags, categories, gaps (or a sharded pointer)
├── manifest.d/                   # per-category shards, only once manifest.yaml crosses its budget
├── requirements/REQ-1.md
├── business-rules/BR-1.md
├── ...                           # one file per object, grouped by id-prefix category
├── source/                       # preserved images from pictures not (yet) turned into an object
                                   # -- not extracted at all, or vision-extraction failed/skipped
└── validation/
    ├── gaps.yaml                 # every skipped/failed/rejected item — never silent
    ├── pending-tags.yaml         # tags awaiting `distl tags review`
    ├── possible-id-matches.yaml  # below-threshold fuzzy matches for human review
    ├── merge-conflicts/*.md      # objects where a human edit and a new extraction disagree
    └── cost-ceiling-alert.md     # present only if the last run halted on the cost ceiling

Every object's frontmatter: id, title, status, priority, description, actors, dependencies, business_rules, constraints, related_apis, confidence, source_location (document/page/section), extraction_method, version, superseded_by, last_updated_by, last_updated_at, extraction_provider, extraction_model, tags.

Design decisions, briefly

Why a manifest instead of a vector index? Real documents compiled so far produce tens of objects (highest observed: 62, on an 18-page document); sharding has been tested against a synthetic 450-object package, not a real one that large yet. At these counts, a flat/sharded YAML index that an agent reads directly is simpler, cheaper, and fully inspectable — no embedding pipeline, no vector store to run. A retrieval-based index is a deliberate future option, not built speculatively ahead of real usage reaching the counts that would justify it.

Why two LLM providers? Any single provider fails sometimes (rate limits, timeouts, content filters). The router retries, then falls back to a second provider, rather than failing the whole chunk — with a hard cost ceiling so a failure storm can't produce an unbounded bill.

Why does a chunk never merge across headings? It used to. A chunk spanning two sections could only cite one (page, section) anchor for every object extracted from it — measured, this produced citations off by as much as 85 pages on a real document. Forcing a chunk boundary at every heading means every citation is exactly the section it came from, at the cost of more (smaller) LLM calls. Traded deliberately: this project treats a wrong citation as a correctness bug, not a cost optimization opportunity.

Why only structural validation so far? Semantic/cross-reference/duplicate/conflict/LLM-review validation each need real failure-rate evidence to justify the added latency and cost before being turned on — turning all of them on speculatively would mean tuning against guesses instead of data. See Not built yet.

Measured, not claimed

Numbers below come from real API calls against real documents (18–143 pages), instrumented by src/distl/telemetry/metrics.py — not projections.

Metric	Result
Cost per document	$0.0016 – $0.1165
Source-location accuracy (post chunking-boundary fix)	27/28 objects (96%) independently verified correct, on a real compile
Source-location accuracy (pre-fix baseline)	4/15 objects (27%) fully correct, across 3 documents
Table-heavy document coverage	1 usable object → 41 objects, after table content stopped being dropped during ingestion

Not measured: whether a compiled package actually makes a coding agent faster or more accurate than handing it the raw document. No consumption-side trial has been run yet — this is the single biggest open question before recommending distl for real use, not just a nice-to-have follow-up metric.

Not built yet

Real gaps, not modesty:

• Validation stages 3–7 (semantic, cross-reference, duplicate, conflict, LLM review) — only stages 1–2 (chunk well-formedness, structural schema) run today.
• Annotation semantics (e.g. strikethrough → deprecated, highlight → important) — not read.
• Diff-aware picture re-extraction on distl update — vision extraction (opt-in, see What distl does) re-attempts every picture on every update call, not just ones in changed sections; text chunks already get this filtering, pictures don't yet.
• Source types beyond PDF/DOCX — Confluence, Jira, Notion, OpenAPI, DB-schema, and Figma adapters exist only as empty placeholder files.
• Multi-document packages — a package is scoped to exactly one source document.
• Consumption-side validation — see the callout above.

None of the above is scheduled on a fixed timeline — each gets built once real usage produces evidence it's worth the added complexity, not built speculatively ahead of that. Track progress or propose one of these via issues.

Development

pip install -e ".[dev]"
pytest

214 tests pass at the time of writing (94% statement coverage), run against real fixture documents with fully mocked/stub provider calls — no network access or API key needed. The "Measured, not claimed" numbers above come from separate, manual runs against real providers, not from the test suite.

Contributing

Issues and PRs are welcome. See CONTRIBUTING.md for dev setup, test/lint commands, and commit/PR conventions — read it before starting anything non-trivial, since this is a single-maintainer project and scope is easiest to agree on before code is written, not after. Participation is governed by CODE_OF_CONDUCT.md. Found a security issue rather than a bug? Report it per SECURITY.md, not as a public issue.

License

MIT — see CHANGELOG.md for what's actually shipped so far.