Find benchmark contamination in LLM training corpora. corpus-assay builds a hashed n-gram index from evaluation benchmarks, streams your Parquet/JSONL shards through a multi-process Rust scanner, and reports which documents overlap which benchmark, down to the individual test item.
It answers one question: does my training data contain text from the benchmarks I am going to report on?
- Fast. The scanner is written in Rust, uses a memory-mapped index, and runs one process per core. That's about 80k short documents/s (28 MB/s of text) on 8 cores; see Performance.
- Attributed. Every flagged document carries per-benchmark hit counts. Reports break leaks down by benchmark, by shard, and by test item.
- Auditable. Indexes and scan runs record the config, the normalization-config hash,
dataset revisions, and the tool version.
reproducere-runs a scan from its manifest after checking that the inputs are unchanged. - Tunable. Built-in reject patterns drop multiple-choice boilerplate. Optional stop-grams can be learned from a background corpus. An item-level gate is available for stricter flagging.
What it detects (and what it doesn't)
A document is flagged when it shares at least min_hits distinct word 13-grams with the
benchmark index and those hits cover at least min_coverage of the document's
distinct 13-grams. With --min-item-coverage C, a document holding at least a share C
of one item's 13-grams is flagged too, however long it is. Matching runs on normalized text (Unicode NFKC, lowercased, stop
words removed), so it survives casing, punctuation, and whitespace changes.
Short items. Items too short to have min_hits 13-grams once stop words are
removed (many short math problems and commonsense multiple-choice questions) can never
be flagged by the n-gram rule; build-index, inspect-index and describe-benchmark
say how many there are. --exact-channel short matches their questions verbatim, after
a light normalization that keeps numbers, operators and stop words (so 2+3 and 2-3
stay different), and reports whether the item's choices appear near the question.
--confirm adds, for every flagged pair, the share of the item's numbers found and the
longest common substring, to tell a copy from another problem built on the same
template. Both are off by default and need an index built by this version.
This is a verbatim-overlap detector. Paraphrases, translations, and heavily edited copies will not be flagged. An experimental spaced-seed channel (docs) recovers lightly edited copies.
Performance
These numbers come from a benchmark run on one GCP c2d-standard-16 VM (AMD EPYC 7B13,
8 physical / 16 logical cores, 63 GB RAM). It scanned 1.56M short documents (0.5 GB of
text, median 28 tokens) in 256 Parquet shards against a benchmark index of 931k
n-grams (22 MB). Each figure is the median of three runs. Peak memory is the summed PSS
of the worker processes.
| Workers | Index backend | Docs/s | MB/s of text | Peak memory |
|---|---|---|---|---|
| 1 | memory | 12,126 | 4.1 | 0.65 GB |
| 8 | memory | 85,487 | 29.3 | 2.30 GB |
| 8 | mmap | 80,357 | 27.5 | 1.91 GB |
| 8 | mmap, with spaced seeds | 54,732 | 18.7 | 4.53 GB |
- Scaling. Throughput grows nearly linearly with physical cores: 7.1× at 8 workers (mmap). Documents/s depends on document length, so MB/s of text is the more portable figure, about 3.5 MB/s per worker.
- Large indexes. Use the mmap backend, the default with more than one worker. It shares one copy of the index across workers. With a 1.6 GB index at 8 workers, peak memory was 2.2 GB with mmap and 29.9 GB with the in-memory backend.
- Contamination rate. Throughput barely changes as contamination rises: from 80.4k to 78.8k docs/s between 0% and 10% contaminated documents.
- Not included. The memory figures exclude the per-item postings sidecar (
.items). Each worker loads it into memory to attribute flagged documents to individual items.
Install
pip install corpus-assay
Prebuilt wheels cover CPython 3.12+ on Linux (x86_64, aarch64), macOS (arm64, x86_64), and Windows (x64). On other platforms pip builds from source, which needs a Rust toolchain.
Quickstart
1. Write a config. Start from the built-in benchmark registry (corpus-assay list-benchmarks shows what's in it):
corpus-assay init-config --benchmarks gsm8k --benchmarks mmlu --out config.yaml
This produces a config you can edit:
benchmarks:
- name: GSM8K
dataset: openai/gsm8k
splits: [test]
revision: 740312add88f781978c0658806c59bc2815b9866 # registry entries are pinned
config_name: main
fields: [question]
label_field: answer # only checked for hidden labels, never indexed
# ...
ngram: 13
min_hits: 3
min_coverage: 0.001
Registry entries index the split and the text an evaluation harness shows the model
(validation, where the test labels are hidden). corpus-assay describe-benchmark <key>
shows an entry's spec and notes.
2. Build the index. This downloads the benchmark splits from the Hugging Face Hub:
corpus-assay build-index --config config.yaml --out-index bench.native
Benchmarks you add without a revision are rejected unless you pass --allow-unpinned.
If a benchmark loads no rows (a removed or gated dataset, a wrong split or path) or
indexes no n-grams, the build fails and writes nothing; --allow-empty-benchmarks
builds the index anyway. The build ends with a per-benchmark table of items and
n-grams, including the items too short for the n-gram rule to flag.
3. Scan your corpus. Shards can be Parquet files or JSONL files, optionally .gz or
.zst compressed. Pass directories, globs, or files:
corpus-assay scan --config config.yaml --index bench.native \
--input "data/**/*.parquet" --id-key id --out-dir scan_out
Records with several texts can be scanned field by field. Each text, and each chat turn, is scanned as its own document, and the records say which field (and role and turn) it was:
# Preference data: prompt and both answers
corpus-assay scan ... --text-key prompt --text-key chosen --text-key rejected
# Chat data: a `messages` column of {role, content} turns
corpus-assay scan ... --messages-key messages --no-split
A shard's string columns that are not scanned are reported in a warning.
4. Report.
corpus-assay report --scan-dir scan_out
Top benchmarks by leak fraction:
GSM8K: 0.11% (85/76252)
Top leaking shards:
shard-0: 85 unique hashes (85.00/1k docs)
Flagged documents are written as records to scan_out/contaminated_docs/*.ndjson. By
default this is a sample of the first 500 flagged per shard; pass --all-records to get
every one:
{"doc_id": "doc500-part-0", "field": "text", "match_count": 85, "src_hits": [[0, 85]],
"item_hits": [{"item_id": 7, "hits": 85, "longest_run_tokens": 75, "unique_hits": 85, ...}]}
One-command variants
# Build (or reuse a cached) index, then scan local shards
corpus-assay run --config config.yaml --input data/ --out-dir scan_out --allow-unpinned
# Same, but scan a Hugging Face dataset split (materialized to Parquet first)
corpus-assay run-hf --config config.yaml --hf-dataset allenai/c4 --split validation \
--hf-revision <commit> --out-dir scan_out --allow-unpinned
Benchmarks from local files
Any datasets loader works, including local JSON or Parquet files. Nothing touches the
network:
benchmarks:
- name: InternalQA
dataset: json
data_files: {test: /data/internal_qa.jsonl}
splits: [test]
fields: [question, answer]
Commands
| Command | Purpose |
|---|---|
init-config |
Write a starter config from registry benchmarks |
list-benchmarks, describe-benchmark |
Browse the built-in benchmark registry |
validate-config |
Validate a config (--check-remote resolves configs and splits on the Hub) |
build-index |
Build the n-gram index (.native + sidecars) |
inspect-index |
Validate an index and list per-benchmark item and n-gram counts |
scan |
Scan Parquet/JSONL shards against an index |
run, run-hf |
Build or reuse a cached index, then scan local shards or an HF split |
filter |
Scan, then write decontaminated copies of the shards with a manifest of the index used |
selftest |
Plant benchmark items into clean text, scan it with your settings, and report recall |
report |
Leak reports (JSON, CSV, Markdown) from a scan directory |
inspect-run |
Validate a scan output directory |
reproduce |
Re-run a scan from its _RUN_MANIFEST.json after checking input hashes |
build-stopgrams-direct |
Learn stop-grams (frequent background n-grams) from a corpus |
build-spaced-index |
Experimental: build the spaced-seed index |
corpus-assay <command> --help shows every option. The full reference is in
docs/cli.md.
Outputs
scan_out/
├── _FINAL_SUMMARY.json / .csv # totals: docs scanned, contaminated (per channel), failed files
├── _RUN_MANIFEST.json # scan config, index file and input hashes, benchmarks, tool versions
├── summaries/<shard>.summary.json # per-shard counts (also the resume cache)
├── per_source_hits/<shard>.hits.bin
├── per_item_best/<shard>.json # best single-document coverage of each item hit
├── contaminated_docs/<shard>.decontam.sample.ndjson # first 500 flagged per shard (--all-records: every one)
└── report/ # written by `report`
├── report.md / report.json
├── leak_by_benchmark.csv / .json
├── leak_by_shard.csv / .json
├── leak_by_item.csv / .json
└── template_variants.csv / .json # with --confirm: likely template variants
Re-running scan with the same inputs and config skips shards that are already done.
File formats are documented in
docs/outputs.md.
Filtering
filter takes the same options as scan, scans every shard (keeping every flagged
record), and writes cleaned copies in the input format, at the inputs' relative
paths, without touching the inputs:
corpus-assay filter --input data/ --index index.native --config config.yaml \
--id-key id --out-dir data_clean/ # drop rows with a flagged part
corpus-assay filter ... --granularity subdoc # drop only the flagged parts
record granularity (the default) drops every row with a flagged sub-document;
subdoc drops the flagged packed parts, empties flagged text fields and removes
flagged chat turns, and drops a row only when nothing is left. Next to the data it
writes _FILTER_MANIFEST.json (the index files' SHA-256, the benchmarks with their
revisions and splits, the per-benchmark counts, gate and thresholds, tool versions,
input and output shard hashes and row counts, and removals per benchmark),
removed_ids.ndjson, a DATASET_CARD_SNIPPET.md to paste into the dataset card,
and the scan itself under _scan/. It refuses a non-empty --out-dir unless
--overwrite.
Checking recall
selftest measures what a scan with your settings would catch. It samples items
from the index, plants them into synthetic filler (or copies of your first
records, --host corpus --input ...), scans the planted file once per gate (and
with the exact channel when --exact-channel is set), and prints recall per gate,
channel and variant, the missed items, and false positives on unplanted hosts:
corpus-assay selftest --index index.native --config config.yaml --out-dir selftest/ \
--n-items 50 --variants verbatim,case,whitespace,wrapper --exact-channel short
Items no enabled channel can match (fewer than min_hits n-grams and not covered
by the exact channel) are listed apart and left out of recall. It exits non-zero
when recall on matchable verbatim plants is below --min-recall (default 1.0), and
writes planted.parquet and truth.jsonl so the result can be reproduced.
Reducing false positives
-
Reject patterns. The bundled normalization config drops n-grams that are multiple-choice scaffolding ("which of the following", "option a", ...). You can supply your own with
corpus-assay --ngram-config my_config.json <command>. The override must be used for bothbuild-indexandscan; a mismatch is detected and refused. -
Stop-grams. Build a list of n-grams that are frequent in a clean background corpus (for example Wikipedia), then pass it with
--stopgrams. Matches on those n-grams are ignored:background: enabled: true corpus: {dataset: wikimedia/wikipedia, config_name: 20231101.en, split: train, fields: [text]}
corpus-assay build-stopgrams-direct --config config.yaml --tau 0.001 \ --out stopgrams.native --max-docs 1000000 corpus-assay scan ... --stopgrams stopgrams.native
-
Item stop-grams. When the protected items repeat an instruction ("Write Python code to solve the problem..."),
stopgrams.items: {max_item_share: 0.01}in the config (orbuild-index --item-stopgrams-share 0.01) drops the n-grams held by more than 1% of the items from the index. -
Item gate.
--gate-mode itemflags a document only if a single benchmark item clears the thresholds on its own, rather than the union of all matches. -
Template variants.
--confirmreports, for each flagged pair, how many of the item's numbers the document has and its longest common substring with the question;reportlists the pairs that look like other problems from the same template.
Python API
do_scan runs its workers with the spawn start method, which re-imports your main
script in every worker. Call it under an if __name__ == "__main__": guard. Without
the guard each worker would re-run the script's setup and the scan; do_scan detects
this and raises a RuntimeError instead of hanging.
from pathlib import Path
from corpus_assay import build_index_hf, do_scan
from corpus_assay.config import (
BuildIndexConfig,
ScanConfig,
load_decontamination_config,
)
from corpus_assay.scanner import OutputLayout, write_run_outputs
if __name__ == "__main__": # required: scan workers are spawned processes
cfg = load_decontamination_config("config.yaml")
build_index_hf(
BuildIndexConfig(
out_path="bench.native",
ngram=cfg.ngram,
benchmarks=cfg.benchmarks,
allow_unpinned=True,
)
)
scan_cfg = ScanConfig(
inputs=["data/"],
text_keys=["text"], # several columns: ["prompt", "chosen", "rejected"]
id_key="id",
index_path="bench.native",
out_dir="scan_out",
ngram=cfg.ngram,
min_hits=cfg.min_hits,
min_coverage=cfg.min_coverage,
)
run = do_scan(scan_cfg)
write_run_outputs(
layout=OutputLayout(Path(scan_cfg.out_dir)), run=run, cfg=scan_cfg
)
print(run.total_contaminated, run.contamination_rate)
Limitations
- At most 64 benchmarks per index, because attribution is stored as a 64-bit mask. Use several indexes for more.
- Text columns must be string columns (
string,large_string, string view or string dictionary); chat columns must be lists of{role, content}structs. - JSONL shards are loaded fully into memory and converted to Parquet before scanning. Use Parquet for very large shards.
- Documents are split on
<|endoftext|>, and each part is scored separately. Records are named<id>-part-<k>(<id>/<field>[/<turn>]-part-<k>when several fields or a messages column are scanned). Pass--no-splitto score each text whole. - Only verbatim overlap is detected; see above.
Documentation
- Configuration reference
- How matching works
- Output and file formats
- CLI reference
- Experimental: spaced seeds
- Architecture
- Relation to the MixtureVitae decontamination
Background
corpus-assay grew out of the decontamination pipeline used for the 13-gram contamination analysis of MixtureVitae (Appendix D). It keeps that pipeline's normalization, hashing and flagging thresholds, but some changes alter results, so default runs do not reproduce the paper's numbers exactly. docs/lineage.md lists the changes and how to get close to the original protocol.
Citing
If you use corpus-assay in your research, please cite it using CITATION.cff (GitHub's "Cite this repository" button). Every release is archived on Zenodo with its own DOI. The concept DOI 10.5281/zenodo.23125328 covers all versions.
Contributing
See CONTRIBUTING.md. Bug reports and pull requests are welcome.
License
Apache-2.0. See LICENSE.
Metadata
Release files for corpus-assay 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| corpus_assay-0.2.0.tar.gz | 471.2 kB | With provenance | Details for corpus_assay-0.2.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| corpus_assay-0.2.0-cp312-abi3-win_amd64.whl | CPython 3.12 | abi3 | Windows x86-64 | Details for corpus_assay-0.2.0-cp312-abi3-win_amd64.whl |
| corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ x86-64 | Details for corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
| corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ ARM64 | Details for corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
| corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl | CPython 3.12 | abi3 | macOS 11.0+ ARM64 | Details for corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl |
| corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl | CPython 3.12 | abi3 | macOS 10.12+ x86-64 | Details for corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl |
Total release size: 19.5 MB
Release files / corpus_assay-0.2.0.tar.gz
| Download URL | corpus_assay-0.2.0.tar.gz |
|---|---|
| Size | 471.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f1f771dba6a3d4fe9b78d56e757eb265737bc605668f4942df88ce6b9c8efc98
|
|
BLAKE2b-256 checksum How to use checksums |
0e47f0ab01615c7684ab24f9a228d2cceb1f09551dbfc1f5ecf6077f47d83adf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / corpus_assay-0.2.0-cp312-abi3-win_amd64.whl
| Download URL | corpus_assay-0.2.0-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 4.0 MB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
26445e16acdef6f968954cc795997a1225ec175b71cd00ca98a0f7f9ac184f50
|
|
BLAKE2b-256 checksum How to use checksums |
79e2c444ddcad2fd4d1aa43624cdd5a4459691a0b9feeda177fc4e12e92b1826
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 4.0 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
1453cc592fac0c1fd7e8031f2c5ccf254f02832e9e3a4776665b4fdd018d2dab
|
|
BLAKE2b-256 checksum How to use checksums |
e842feaaec73b5b3088e629e0bcf3570f788a94092294cbe9c9394795ed8cf6c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
3a9bebf94d2367e73367fa08861ab1494050894c45744baab2bac7616650eaad
|
|
BLAKE2b-256 checksum How to use checksums |
c3dd595604a2f140dfb8647c424e22bcb04a16984dcb2ef1e2597c8fe1a56619
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.5 MB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
184658fccc364597d4c09427e378a477f027976998c00fb420ad7add4082ec63
|
|
BLAKE2b-256 checksum How to use checksums |
845f424a1ece185f366de7ae8e77d37fb375064e8651d77b61ea7049c8bfab65
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logRelease files / corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl
| Download URL | corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 3.8 MB |
| Tags | CPython 3.12 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
8ed67b3974362f446bacaa71ad052cfc7919fe5e01ce3a8486ab81f8a757b382
|
|
BLAKE2b-256 checksum How to use checksums |
0f5c4a0d112ce55d797b1128b00b71f7c4ba2c0296bc0a1849331a115b01a16f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logSecurity for corpus-assay 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 6 files in this release have been uploaded with provenance metadata.
Attestation for corpus_assay-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.