Quarry

A config-driven knowledge-ingestion harness — a CLI that owns the deterministic half of turning a source (a YouTube video, a web page, …) into a linked article in a markdown knowledge wiki.

Quarry resolves a source adapter, fetches, writes immutable raw material, hands a machine-readable compile-spec to an LLM (or human) for the one irreducibly-generative step — writing the article — then verifies provenance, lints, and commits. It never calls a model, so it is runnable and testable without API keys.

You extract raw material from the world and refine it into something structured. That is the pipeline, and the name.

Why

The deterministic mechanics of disciplined knowledge ingestion — fetch, immutable raw, a verifiable manifest, provenance checking, structural lint, semantic link discovery — are worth doing the same way every time. Quarry packages exactly that machinery and nothing else: it is not an LLM wrapper, not a note-taking app, and not opinionated about your wiki's shape. Every convention is configuration.

Install

pip install quarry-kb[all]          # all shipped adapters
pip install quarry-kb               # core only (PyYAML); add adapters as extras

Adapters (each an extra): youtube (transcript + Whisper fallback), web, github (gitingest), pdf (PyMuPDF4LLM + OCR), instagram, plus whisper for local audio transcription. Adapters extract raw content deterministically — no LLM. The core needs no API key; a few adapters take optional credentials (e.g. Instagram cookies for private posts) — see each adapter's module docstring for setup.

No Reddit adapter (removed in 0.4.0). Reddit became a losing battle for a deterministic, no-auth fetcher: it fingerprint-blocks pure-Python clients via TLS/JA3 (so it needed curl_cffi impersonation just to get a 200), throttles by IP reputation with no Retry-After (so back-off is guesswork), and its Nov-2025 Responsible Builder Policy gates all API apps — including read-only OAuth — behind manual approval. The net result was a fragile, perpetually-degrading path that violated Quarry's "runnable without keys, deterministically" principle. Rather than ship something that mostly doesn't work, it's gone. To ingest a Reddit thread, save the page (or its .json) and feed it through the web/pdf adapter, or add your own adapter via the entry-point hook. (curl_cffi lives on — the instagram extra uses it for yt-dlp's impersonation.)

Installed as quarry-kb on PyPI (the bare name was taken); the command and import are both quarry.

Requires Python 3.11+.

60-second quickstart

cd your-wiki-repo
quarry init                  # scaffold a documented quarry.toml + gitignore .quarry/
# edit quarry.toml to match your wiki's conventions

quarry ingest <url>          # fetch -> raw/ + compile-manifest, then STOP
# ...now write the wiki article using the printed compile-spec...
quarry finish <slug>         # verify provenance -> lint -> commit (no push unless --push)

Other commands: quarry lint, quarry adapters, quarry related <article>, quarry densify [--apply], quarry doctor.

The two-call seam is deliberate: ingest and finish are separate processes, and the generative step (writing the article) happens between them — done by you or your agent, never by Quarry.

Configuration reference

Quarry is configured by a quarry.toml discovered by walking up from the current directory (a [tool.quarry] table in pyproject.toml is a fallback). quarry init writes a fully-commented default. Tables:

See examples/quarry.toml for the annotated default.

Authoring an adapter

A new source type is one small adapter discoverable via the quarry.adapters entry-point group — no fork. See CONTRIBUTING.md for the contract, the hermetic-test requirement, and a worked pdf example.

Status

Pre-1.0, but feature-complete for v1 and fully tested. See SPEC.md for the design and ISA.md for the ideal-state criteria and build provenance.

License

MIT — see LICENSE.