chunk-bench

Benchmark RAG chunking strategies on your own documents. Compare fixed, sliding, paragraph, and recursive chunking with real retrieval metrics.

Why chunk-bench?

Chunking strategy is one of the highest-impact decisions in a RAG pipeline — 2026 benchmarks show up to a 9% recall gap between best and worst strategy on the same corpus. But most teams pick a strategy once and never measure whether it's actually working for their documents.

chunk-bench lets you run all four strategies against your own text and queries in one call, and get back recall, precision, and MRR scores — no embeddings required, no external services.

Install

pip install chunk-bench

Quick start

from chunk_bench import ChunkBench

bench = ChunkBench()

report = bench.run(
    text=your_document_text,
    queries=["What is retrieval augmented generation?",
             "How does chunking affect retrieval quality?",
             "What is the difference between fixed and semantic chunking?"],
)

print(report.summary_table())

Output:

Strategy           Chunks  Avg Tokens   Recall  Precision     MRR      F1
────────────────────────────────────────────────────────────────────────
recursive               18         94    0.857      0.733   0.833   0.790
paragraph               12        142    0.810      0.700   0.810   0.752
fixed                   24         71    0.762      0.667   0.762   0.711
sliding                 31         55    0.714      0.633   0.714   0.671
────────────────────────────────────────────────────────────────────────

  Best recall:    recursive (0.857)
  Best F1:        recursive (0.790)
  Best MRR:       recursive (0.833)

Providing your own relevance terms

report = bench.run(
    text=your_document,
    queries=["What is GDPR?", "What are data subject rights?"],
    relevant_terms=[
        ["GDPR", "General Data Protection", "regulation"],
        ["subject", "rights", "access", "erasure", "portability"],
    ],
    top_k=5,
)

Use specific strategies

report = bench.run(
    text=text,
    queries=queries,
    strategies=["fixed", "recursive"],  # skip sliding and paragraph
    chunk_size=512,
    overlap=50,
)

Chunk any text directly

from chunk_bench import chunk

chunks = chunk(text, strategy="recursive", chunk_size=512, overlap=50)
for c in chunks:
    print(f"[{c.index}] ~{c.token_count} tokens: {c.text[:60]}")

CLI

chunk-bench document.txt --queries "What is X?" "How does Y work?" --strategies fixed recursive --json

Strategies

Linda Oraegbunam | LinkedIn | Twitter | GitHub