chunkshop (Python)

Reference implementation of the chunkshop ingest tool. v0.5.0 beta.

New here? Start with the end-to-end tutorial — a guided walkthrough from zero (no Postgres) to a running semantic query.

This file is the field-by-field reference: every CLI flag, every YAML field, the troubleshooting table. Use it alongside the tutorial once you know what you're doing.

For the high-level shape and mermaid diagram, see the top-level README.

Install

From source when you want sample corpora, dev tooling, or unreleased branch work:

git clone https://github.com/yonk-labs/chunkshop && cd chunkshop/python
uv sync --extra dev --extra all-backends

Published package:

pip install 'chunkshop[all-backends]'

As a path dependency from another project:

[tool.uv.sources]
chunkshop = { path = "../chunkshop/python", editable = true }

Optional extras:

Python ≥ 3.12 required.

Prerequisites

• One supported target backend: Postgres ≥ 14 + pgvector, MariaDB 11.7+, SQLite + sqlite-vec, or ClickHouse 24.10+.
• For Postgres, CREATE EXTENSION vector; must succeed in your target DB.
• Disk space for model cache in ~/.cache/fastembed/ — ~85 MB for int8 bge-base, ~550 MB for nomic.
• An env var holding your DSN. The target config references it by name, not by value.

Quick run

export CHUNKSHOP_DSN="postgresql://postgres:postgres@localhost:5432/mydb"

# Point at the sample corpus in docs/samples/ for a real end-to-end run:
chunkshop ingest --config ../docs/samples/sample.yaml

# Or copy the template and edit it:
cp src/chunkshop/configs/example-files-to-bge.yaml my-cell.yaml
chunkshop ingest --config my-cell.yaml

Success looks like:

{
  "cell_name": "example_files",
  "docs_processed": 47,
  "chunks_written": 312,
  "wall_seconds": 18.4,
  "error": null
}

CLI

Two subcommands: ingest (one cell) and orchestrate (many cells in parallel).

chunkshop ingest

Runs one YAML end-to-end.

chunkshop ingest --config PATH [--doc-limit N] [--log PATH] [--omp-threads N]

Exit code: 0 on success, 1 if the cell errored. Stdout = a JSON summary.

chunkshop orchestrate

Runs N cells in parallel as subprocesses.

chunkshop orchestrate (--config-dir DIR | --config PATH [--config PATH ...])
                      [--concurrency N]
                      [--checkpoints "60,120,300,600"]
                      [--timeout SECONDS]
                      [--smoke | --full]

Stdout = checkpoint reports during the run, JSON summary at the end.

chunkshop bakeoff

Runs a chunker × embedder matrix against a corpus with hand-written gold queries, scores recall@k + MRR per combo, writes a leaderboard + a runnable recommended.yaml. Config-driven — the matrix lives in YAML, not on the command line.

chunkshop bakeoff --config PATH [--dsn DSN] [--yes] [--keep-schema]

Outputs land in skill-output/bakeoff/{name}/:

• results.json — raw per-combo + per-query data.
• report.md — leaderboard sorted by MRR, per-query detail, statistical- power caveat.
• recommended.yaml — top combo pre-filled as a runnable chunkshop ingest cell.

Full walkthrough: ../docs/tutorial-bakeoff.md. Recipe card: ../docs/quickstart-bakeoff.md.

YAML reference

Every cell config has five sections plus an optional runtime. Extra keys are rejected (extra="forbid" in pydantic), so typos fail loudly.

cell_name: my_cell
source:   { ... }
chunker:  { ... }
embedder: { ... }
extractor: { ... }   # optional, defaults to {type: none}
target:   { ... }
runtime:  { ... }    # optional, sensible defaults below

source

chunker

Seven chunkers in three families. Pick one per cell.

Structural — split on headings, paragraphs, or word counts:

Semantic — splits on embedding-drift boundaries (no heading needed):

Pass boundary_model: "same" to reuse the cell's main embedder (trades speed for memory). See ../docs/tutorial-semantic.md.

Summary-layer — wrap any base chunker and change what gets embedded vs. what gets stored (summary_embed) or emit fine+coarse rows linked by group_id (hierarchical_summary):

The summarizer config is a discriminated union: {mode: external, field: ...} pulls a pre-computed summary from a source document metadata field; {mode: callable, module: "lede.tfidf", function: "summarize", kwargs: {...}} imports lazily at first use; {mode: passthrough} reuses the raw chunk as the summary (baseline). See ../docs/summaries.md and ../docs/tutorial-summaries.md.

Full per-chunker guidance: ../docs/chunkers.md.

embedder

Only fastembed today.

extractor

RAKE downloads NLTK corpora (stopwords, punkt) on first use to ~/nltk_data/.

target

target.documents (Python/Postgres only)

For Postgres targets in the Python implementation, target.documents.enabled: true writes a companion one-row-per-document table next to the chunk table. It is off by default for compatibility.

target:
  type: postgres
  dsn_env: CHUNKSHOP_DSN
  database: mydata
  table: chunks
  documents:
    enabled: true
    table: documents
    store_full_content: true
    store_lede_report: true

Current support boundary:

• Python + Postgres: supported.
• Python + SQLite/MariaDB/ClickHouse: rejected at config load.
• Rust: rejected at config load when documents.enabled: true; Rust does not write the companion document table yet.

Multi-source ingest

Multiple cells can write to the same table by tagging each cell's rows with a source_tag. Cell A creates the table with mode: create_if_missing, Cell B appends with mode: append and its own tag. Queries filter or group by the source column. See ../docs/tutorial-multi-source.md for the end-to-end walkthrough.

target:
  dsn_env: CHUNKSHOP_DSN
  schema: mydata
  table: all_docs
  mode: append
  source_tag: support_tickets

runtime

Environment variables

Troubleshooting

"no files matched glob: /path/**/*.md"

Your source.glob didn't match anything. Test it in a shell first:

ls /path/**/*.md | head

Note that chunkshop uses Python's glob.glob(..., recursive=True) — ** only matches across directories when it's its own path component (/foo/**/*.md, not /foo/**.md).

"relation already exists" on second run

target.overwrite is false by default. Either flip it to true (drops + recreates) or drop the table yourself. The ON CONFLICT DO UPDATE in the writer will also happily upsert into an existing table.

"model X produced dim Y, config says dim=Z"

Your YAML's embedder.dim doesn't match the model's output. Look up the right dim in ../docs/embedders.md — bge-small=384, bge-base=768, nomic=768.

"CREATE EXTENSION IF NOT EXISTS vector" fails with permission denied

Your DB role can't create extensions. Ask a superuser to run it once per database:

CREATE EXTENSION IF NOT EXISTS vector;

Then re-run chunkshop — the sink's CREATE EXTENSION IF NOT EXISTS will be a no-op.

"table/schema must match ^[a-z_][a-z0-9_]*$"

chunkshop refuses to interpolate mixed-case or quoted identifiers — SQL injection safety via allowlist. Lowercase your schema and table.

Ingest is slow and my CPU fans are loud

Three knobs. Pick one:

• Drop embedder.batch_size from 64 to 32 — less memory pressure, slower per-doc.
• Set embedder.threads: 4 (or 2) — caps ORT's worker pool.
• If running under orchestrate, reduce --concurrency.

See the thread-tuning table in ../docs/embedders.md.

First run hangs on "downloading model"

Fastembed is pulling the ONNX from HuggingFace. Network / HF outage. Check curl -sI https://huggingface.co/ and your proxy settings. The file lands in ~/.cache/fastembed/<model-name>/.

nltk errors on first rake_keywords run

The extractor downloads stopwords, punkt, punkt_tab into ~/nltk_data/ on first use. Behind a strict firewall? Pre-download once:

import nltk
for r in ("stopwords", "punkt", "punkt_tab"):
    nltk.download(r)

Using chunkshop as a library

from chunkshop import load_config
from chunkshop.runner import run_cell

cfg = load_config("my-cell.yaml")
result = run_cell(cfg)
print(result.docs_processed, result.chunks_written, result.wall_seconds)

Or skip the YAML and build a CellConfig directly — every section is a plain pydantic model.

Tests

cd python
uv run pytest

Most tests are offline. test_embedder_fastembed.py and test_int8_registry.py download the int8 bge-base model on first run and cache it — budget ~85 MB + a few seconds the first time.