Skip to main content

MTCB Logo

🔬 mtcb ✨

The benchmark for evaluating chunking strategies in RAG pipelines.

PyPI version License GitHub stars Downloads

Installation • Quick Start • Benchmarks • Usage • Metrics

MTCB (Massive Text Chunking Benchmark) is a standardized evaluation framework for text chunking in RAG systems. It measures how well your chunking and embedding strategy retrieves relevant passages across 9 diverse domains, from legal contracts to scientific papers. Built on top of Chonkie.

📦 Installation

pip install mtcb

🚀 Quick Start

Run the lightweight nano benchmark to evaluate a chunking strategy in minutes:

from mtcb import NanoBenchmark
from chonkie import RecursiveChunker

benchmark = NanoBenchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

🧩 Available Benchmarks

Full Benchmark

The full MTCB benchmark spans 9 domains with ~17k questions across ~3k documents:

Dataset Domain Documents Questions
🧸 Gacha Classic Literature (Gutenberg) 100 2,878
💼 Ficha SEC Financial Filings 88 1,331
📝 Macha GitHub READMEs 445 1,812
💻 Cocha Multilingual Code 1,000 2,372
📊 Tacha Financial Tables (TAT-QA) 349 2,065
🔬 Sencha Scientific Papers (QASPER) 243 1,507
⚖️ Hojicha Legal Contracts (CUAD) 194 1,568
🏥 Ryokucha Medical Guidelines (NICE/CDC/WHO) 241 1,351
🎓 Genmaicha MIT OCW Lecture Transcripts 250 2,037
Total 2,910 16,921

Nano Benchmark

For fast iteration during development, MTCB provides a lightweight nano benchmark with ~100 questions per dataset. Documents are selected to maximize question density:

Dataset Domain Documents Questions
🧸 nano-gacha Classic Literature 5 100
💼 nano-ficha SEC Financial Filings 5 100
📝 nano-macha GitHub READMEs 19 100
💻 nano-cocha Multilingual Code 26 100
📊 nano-tacha Financial Tables 11 100
🔬 nano-sencha Scientific Papers 13 100
⚖️ nano-hojicha Legal Contracts 10 100
🏥 nano-ryokucha Medical Guidelines 12 100
🎓 nano-genmaicha Lecture Transcripts 7 100
Total 108 900

🔧 Usage

MTCB works with Chonkie — any chunker that extends chonkie.BaseChunker is supported out of the box.

Full Benchmark

Run the complete benchmark across all 9 domains:

from mtcb import Benchmark
from chonkie import RecursiveChunker

benchmark = Benchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

Individual Evaluators

Run a single domain-specific evaluator:

from mtcb import GachaEvaluator
from chonkie import RecursiveChunker

evaluator = GachaEvaluator(
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
    cache_dir="./cache"
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Custom Datasets

Evaluate on your own corpus using SimpleEvaluator:

from mtcb import SimpleEvaluator
from chonkie import RecursiveChunker

evaluator = SimpleEvaluator(
    corpus=["Your document text here...", "Another document..."],
    questions=["What is X?", "How does Y work?"],
    relevant_passages=["passage that must be in retrieved chunk", "another passage"],
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Dataset Generation

Generate verified QA datasets from your own documents:

from mtcb import DatasetGenerator

generator = DatasetGenerator(deduplicate=True)
result = generator.generate(
    corpus=["Your document text..."],
    samples_per_document=10,
    output_path="./output.jsonl",
)

print(f"Generated {result.total_verified} verified samples")
for sample in result.samples:
    print(f"Q: {sample.question}")
    print(f"A: {sample.answer}")

📊 Metrics

MTCB evaluates retrieval quality using:

  • Recall@k: Percentage of questions where the relevant passage appears in the top-k results
  • Precision@k: Ratio of relevant chunks in the top-k results
  • MRR@k: Mean Reciprocal Rank — how high the first relevant result ranks
  • NDCG@k: Normalized Discounted Cumulative Gain — position-weighted relevance scoring

📚 Citation

If you use MTCB in your research, please cite:

@software{mtcb2025,
  author = {Bhavnick Minhas and Shreyash Nigam},
  title = {MTCB: Massive Text Chunking Benchmark},
  url = {https://github.com/chonkie-inc/mtcb},
  version = {0.1.0},
  year = {2025},
}

Metadata

Release files for mtcb 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mtcb 0.1.0
File Size Uploaded Provenance Actions
mtcb-0.1.0.tar.gz 37.8 kB Without provenance Details for mtcb-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mtcb 0.1.0
File Interpreter ABI Platform Actions
mtcb-0.1.0-py3-none-any.whl Python 3 none any Details for mtcb-0.1.0-py3-none-any.whl

Total release size: 85.1 kB

Release files / mtcb-0.1.0.tar.gz

Download URL mtcb-0.1.0.tar.gz
Size 37.8 kB
Tags Source
SHA-256 checksum
How to use checksums
5d2175203845db3d66030aabbffbe0dd9dc831a5c32800f8e9a31b552b378917
BLAKE2b-256 checksum
How to use checksums
684f47ea337c09fdb6ae3aa976bb3ab035f51b6466fea3a750b3b76110f33254
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / mtcb-0.1.0-py3-none-any.whl

Download URL mtcb-0.1.0-py3-none-any.whl
Size 47.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f0f9cfea1c34714530db4bdf0cc303527801cd2c8043e4362f6870cec862c675
BLAKE2b-256 checksum
How to use checksums
2d677c4618ef8c3f4d28a921404e8a4a85294b6f0cfe54915cc2668e8d2f6cce
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for mtcb 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page