chunk-bench 1.0.0

Benchmark RAG chunking strategies on your own documents. Compare fixed, sliding, paragraph, recursive, and semantic chunking with real retrieval metrics.

chunk-bench

Benchmark RAG chunking strategies on your own documents. Compare fixed, sliding, paragraph, and recursive chunking with real retrieval metrics.

---

Why chunk-bench?

Chunking strategy is one of the highest-impact decisions in a RAG pipeline — 2026 benchmarks show up to a 9% recall gap between best and worst strategy on the same corpus. But most teams pick a strategy once and never measure whether it's actually working for their documents.

chunk-bench lets you run all four strategies against your own text and queries in one call, and get back recall, precision, and MRR scores — no embeddings required, no external services.

---

Install

pip install chunk-bench

---

Quick start

from chunk_bench import ChunkBench

bench = ChunkBench()

report = bench.run(
    text=your_document_text,
    queries=["What is retrieval augmented generation?",
             "How does chunking affect retrieval quality?",
             "What is the difference between fixed and semantic chunking?"],
)

print(report.summary_table())

Output:

Strategy           Chunks  Avg Tokens   Recall  Precision     MRR      F1
────────────────────────────────────────────────────────────────────────
recursive               18         94    0.857      0.733   0.833   0.790
paragraph               12        142    0.810      0.700   0.810   0.752
fixed                   24         71    0.762      0.667   0.762   0.711
sliding                 31         55    0.714      0.633   0.714   0.671
────────────────────────────────────────────────────────────────────────

  Best recall:    recursive (0.857)
  Best F1:        recursive (0.790)
  Best MRR:       recursive (0.833)

---

Providing your own relevance terms

report = bench.run(
    text=your_document,
    queries=["What is GDPR?", "What are data subject rights?"],
    relevant_terms=[
        ["GDPR", "General Data Protection", "regulation"],
        ["subject", "rights", "access", "erasure", "portability"],
    ],
    top_k=5,
)

---

Use specific strategies

report = bench.run(
    text=text,
    queries=queries,
    strategies=["fixed", "recursive"],  # skip sliding and paragraph
    chunk_size=512,
    overlap=50,
)

---

Chunk any text directly

from chunk_bench import chunk

chunks = chunk(text, strategy="recursive", chunk_size=512, overlap=50)
for c in chunks:
    print(f"[{c.index}] ~{c.token_count} tokens: {c.text[:60]}")

---

CLI

chunk-bench document.txt --queries "What is X?" "How does Y work?" --strategies fixed recursive --json

---

Strategies

Strategy	Description	Best for
fixed	Split at regular character intervals with overlap	Simple, uniform documents
sliding	Overlapping windows stepping forward	When context preservation matters most
paragraph	Split on double newlines, merge small paragraphs	Structured documents with clear sections
recursive	Try paragraph → sentence → word boundaries	Most document types — good default

---

Linda Oraegbunam | LinkedIn | Twitter | GitHub