chunklab

Find out which chunking strategy actually retrieves your answers best — on your own documents — in 60 seconds.

When a RAG system gives a wrong answer, the failure is often in retrieval — the passage that holds the answer never reaches the model — not in the LLM's reasoning over what it was given. And one of the most overlooked variables in retrieval is chunking: how you split documents before embedding them. Independent benchmarks that hold the embedding model and retriever fixed and vary only the chunking find that it moves retrieval quality on the same corpus (see, e.g., Chroma's Evaluating Chunking Strategies for Retrieval).

On the example corpus bundled with this repo, the choice of strategy moves per-document recall@5 by up to 28 points (API-reference document: 0.84 best vs 0.56 worst strategy), and three different strategies win depending on the document — reproduce both claims with chunklab run --docs examples/corpus --questions examples/questions.yaml (enforced in CI by tests/test_corpus_discriminates.py).

There is no universal best strategy: which one wins depends on your documents and your questions. chunklab makes that a one-command, fully-local, evidence-based decision instead of a guess.

What it does

You give it your documents and a handful of questions (each tagged with the "gold" passage that answers it). It runs several chunking strategies, indexes and retrieves for each, and tells you which strategy actually retrieves your answer-bearing text best — and why.

docs + questions ─▶ [fixed · recursive · semantic · structure] ─▶ ranked report + diagnostics

• Runs fully locally. Default embeddings are a small local model (BAAI/bge-small-en-v1.5) — no API key, your documents never leave your machine.
• Explains itself. Per-strategy diagnostics (token distribution, % tiny fragments, boundary health, table integrity) tell you why a strategy won or lost.
• Three outputs: a console table, a standalone HTML report (per-question drill-down + chunk-boundary visualization), and a machine-readable JSON report for CI.

Install

pip install chunklab

The first run downloads the embedding model (~130 MB) once and caches it.

60-second quickstart

# clone the repo to get the example corpus, or point --docs at your own files
chunklab run --docs examples/corpus --questions examples/questions.yaml

Console output of exactly that command (regenerated from a real run, never hand-edited):

ChunkLab — 5 document(s), 129 scored questions, top_k=5, model=BAAI/bge-small-en-v1.5

 Strategy            ┃ recall@5 ┃ hit@5 ┃  MRR ┃ #chunks ┃ med_tok ┃ %tiny ┃ boundary 
━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━
 ▶ structure         │     0.81 │  0.81 │ 0.63 │      71 │     346 │   25% │     100% 
   fixed             │     0.79 │  0.79 │ 0.61 │      61 │     512 │    2% │       2% 
   recursive         │     0.78 │  0.78 │ 0.59 │      62 │     458 │    5% │     100% 
   semantic          │     0.72 │  0.72 │ 0.55 │      47 │     592 │    2% │     100% 
   semantic_no_floor │     0.70 │  0.70 │ 0.53 │      59 │     470 │   24% │     100% 

Recommendation:
  Use STRUCTURE chunking (max_tokens=800). It gave the best retrieval on your corpus (recall@5 = 
0.81). Note: 27% of structure chunks exceed the embedding model's max sequence length and may be 
truncated at embed time.

Open chunklab_report/report.html for the full drill-down and the chunk-boundary visualization. The example corpus itself is documented in examples/CORPUS.md — five documents designed so that different strategies win on different documents.

Writing your questions.yaml

The tool scores retrieval offline by checking whether a gold snippet — a verbatim (or near-verbatim) passage from your document that answers the question — lands inside a retrieved chunk. Aim for 10–20 questions.

questions:
  - id: q1
    query: "What is the termination notice period?"
    gold_snippets:
      - "written notice at least 30 days prior to termination"
    tags: [contracts]
  - id: q2
    query: "How is overtime compensated?"
    gold_snippets:
      - "Overtime is paid at 1.5x the regular hourly rate"
  - id: q3
    query: "What is the dress code?"
    # no gold_snippets -> skipped (with a warning), so scoring stays deterministic

Tips:

• Copy the gold snippet verbatim from the source so matching is reliable (small drift is absorbed by fuzzy matching, threshold configurable).
• A question with no gold_snippets is skipped in the MVP — add the passage to include it.

Configuration

chunklab run works with no config. To customize, pass --config config.yaml:

embedding:
  backend: local                 # local (default) | openai (coming soon)
  model: BAAI/bge-small-en-v1.5
retrieval:
  mode: dense                    # dense (bm25/hybrid coming soon)
  top_k: 5
eval:
  fuzzy_threshold: 0.90
  ranking_metric: recall_at_k    # recall_at_k | mrr | hit_rate_at_k
  min_floor_tokens: 200
strategies:
  - { name: fixed,     params: { chunk_size: 512, overlap: 64 } }
  - { name: recursive, params: { chunk_size: 512, overlap: 64 } }
  - { name: semantic,  params: { breakpoint_percentile: 95, min_tokens: 200, max_tokens: 1000 } }
  - { name: structure, params: { max_tokens: 800 } }
output:
  formats: [console, html, json]
  dir: ./chunklab_report

See examples/config.example.yaml for the full default.

The chunking strategies

Run chunklab strategies to list them.

Python API

from chunklab import evaluate

report = evaluate(
    docs="./docs",
    questions="./questions.yaml",
    config=None,  # or a path / a Config object
)
print(report.recommendation)
for r in report.strategy_results:      # ranked best-first
    print(r.strategy, r.recall_at_k, r.mrr)

The EvalReport schema is a versioned public contract (report.schema_version, documented field-by-field in docs/schema.md) — serialize it with report.model_dump_json() for CI or dashboards.

Web demo

pip install "chunklab[demo]"
chunklab demo          # launches a local Gradio app

Upload a document, type a few questions with gold snippets, pick strategies, and see the ranked comparison plus the chunk visualization.

What this is (and isn't)

It is a fast, focused, local pre-flight utility that answers exactly one question: which chunking strategy retrieves best on my corpus, and why?

It is not a production RAG framework, a vector database, a document parser, or a general LLM-eval platform. It deliberately does one thing well.

Supported inputs

Documents: PDF, DOCX, TXT, MD. Corpora of tens to low-hundreds of documents (this is a pre-flight tool, not a batch pipeline).

License

MIT — permissive on purpose. Runs locally, no telemetry, no phone-home.