Skip to main content

corpus-assay: audit benchmark overlap

CI PyPI Python versions License DOI

Find benchmark contamination in LLM training corpora. corpus-assay builds a hashed n-gram index from evaluation benchmarks, streams your Parquet/JSONL shards through a multi-process Rust scanner, and reports which documents overlap which benchmark, down to the individual test item.

It answers one question: does my training data contain text from the benchmarks I am going to report on?

  • Fast. The scanner is written in Rust, uses a memory-mapped index, and runs one process per core. That's about 80k short documents/s (28 MB/s of text) on 8 cores; see Performance.
  • Attributed. Every flagged document carries per-benchmark hit counts. Reports break leaks down by benchmark, by shard, and by test item.
  • Auditable. Indexes and scan runs record the config, the normalization-config hash, dataset revisions, and the tool version. reproduce re-runs a scan from its manifest after checking that the inputs are unchanged.
  • Tunable. Built-in reject patterns drop multiple-choice boilerplate. Optional stop-grams can be learned from a background corpus. An item-level gate is available for stricter flagging.

What it detects (and what it doesn't)

A document is flagged when it shares at least min_hits distinct word 13-grams with the benchmark index and those hits cover at least min_coverage of the document's distinct 13-grams. With --min-item-coverage C, a document holding at least a share C of one item's 13-grams is flagged too, however long it is. Matching runs on normalized text (Unicode NFKC, lowercased, stop words removed), so it survives casing, punctuation, and whitespace changes.

Short items. Items too short to have min_hits 13-grams once stop words are removed (many short math problems and commonsense multiple-choice questions) can never be flagged by the n-gram rule; build-index, inspect-index and describe-benchmark say how many there are. --exact-channel short matches their questions verbatim, after a light normalization that keeps numbers, operators and stop words (so 2+3 and 2-3 stay different), and reports whether the item's choices appear near the question. --confirm adds, for every flagged pair, the share of the item's numbers found and the longest common substring, to tell a copy from another problem built on the same template. Both are off by default and need an index built by this version.

This is a verbatim-overlap detector. Paraphrases, translations, and heavily edited copies will not be flagged. An experimental spaced-seed channel (docs) recovers lightly edited copies.

Performance

These numbers come from a benchmark run on one GCP c2d-standard-16 VM (AMD EPYC 7B13, 8 physical / 16 logical cores, 63 GB RAM). It scanned 1.56M short documents (0.5 GB of text, median 28 tokens) in 256 Parquet shards against a benchmark index of 931k n-grams (22 MB). Each figure is the median of three runs. Peak memory is the summed PSS of the worker processes.

Workers Index backend Docs/s MB/s of text Peak memory
1 memory 12,126 4.1 0.65 GB
8 memory 85,487 29.3 2.30 GB
8 mmap 80,357 27.5 1.91 GB
8 mmap, with spaced seeds 54,732 18.7 4.53 GB
  • Scaling. Throughput grows nearly linearly with physical cores: 7.1× at 8 workers (mmap). Documents/s depends on document length, so MB/s of text is the more portable figure, about 3.5 MB/s per worker.
  • Large indexes. Use the mmap backend, the default with more than one worker. It shares one copy of the index across workers. With a 1.6 GB index at 8 workers, peak memory was 2.2 GB with mmap and 29.9 GB with the in-memory backend.
  • Contamination rate. Throughput barely changes as contamination rises: from 80.4k to 78.8k docs/s between 0% and 10% contaminated documents.
  • Not included. The memory figures exclude the per-item postings sidecar (.items). Each worker loads it into memory to attribute flagged documents to individual items.

Install

pip install corpus-assay

Prebuilt wheels cover CPython 3.12+ on Linux (x86_64, aarch64), macOS (arm64, x86_64), and Windows (x64). On other platforms pip builds from source, which needs a Rust toolchain.

Quickstart

1. Write a config. Start from the built-in benchmark registry (corpus-assay list-benchmarks shows what's in it):

corpus-assay init-config --benchmarks gsm8k --benchmarks mmlu --out config.yaml

This produces a config you can edit:

benchmarks:
  - name: GSM8K
    dataset: openai/gsm8k
    splits: [test]
    revision: 740312add88f781978c0658806c59bc2815b9866   # registry entries are pinned
    config_name: main
    fields: [question]
    label_field: answer   # only checked for hidden labels, never indexed
  # ...
ngram: 13
min_hits: 3
min_coverage: 0.001

Registry entries index the split and the text an evaluation harness shows the model (validation, where the test labels are hidden). corpus-assay describe-benchmark <key> shows an entry's spec and notes.

2. Build the index. This downloads the benchmark splits from the Hugging Face Hub:

corpus-assay build-index --config config.yaml --out-index bench.native

Benchmarks you add without a revision are rejected unless you pass --allow-unpinned. If a benchmark loads no rows (a removed or gated dataset, a wrong split or path) or indexes no n-grams, the build fails and writes nothing; --allow-empty-benchmarks builds the index anyway. The build ends with a per-benchmark table of items and n-grams, including the items too short for the n-gram rule to flag.

3. Scan your corpus. Shards can be Parquet files or JSONL files, optionally .gz or .zst compressed. Pass directories, globs, or files:

corpus-assay scan --config config.yaml --index bench.native \
  --input "data/**/*.parquet" --id-key id --out-dir scan_out

Records with several texts can be scanned field by field. Each text, and each chat turn, is scanned as its own document, and the records say which field (and role and turn) it was:

# Preference data: prompt and both answers
corpus-assay scan ... --text-key prompt --text-key chosen --text-key rejected
# Chat data: a `messages` column of {role, content} turns
corpus-assay scan ... --messages-key messages --no-split

A shard's string columns that are not scanned are reported in a warning.

4. Report.

corpus-assay report --scan-dir scan_out
Top benchmarks by leak fraction:
  GSM8K: 0.11% (85/76252)
Top leaking shards:
  shard-0: 85 unique hashes (85.00/1k docs)

Flagged documents are written as records to scan_out/contaminated_docs/*.ndjson. By default this is a sample of the first 500 flagged per shard; pass --all-records to get every one:

{"doc_id": "doc500-part-0", "field": "text", "match_count": 85, "src_hits": [[0, 85]],
 "item_hits": [{"item_id": 7, "hits": 85, "longest_run_tokens": 75, "unique_hits": 85, ...}]}

One-command variants

# Build (or reuse a cached) index, then scan local shards
corpus-assay run --config config.yaml --input data/ --out-dir scan_out --allow-unpinned

# Same, but scan a Hugging Face dataset split (materialized to Parquet first)
corpus-assay run-hf --config config.yaml --hf-dataset allenai/c4 --split validation \
  --hf-revision <commit> --out-dir scan_out --allow-unpinned

Benchmarks from local files

Any datasets loader works, including local JSON or Parquet files. Nothing touches the network:

benchmarks:
  - name: InternalQA
    dataset: json
    data_files: {test: /data/internal_qa.jsonl}
    splits: [test]
    fields: [question, answer]

Commands

Command Purpose
init-config Write a starter config from registry benchmarks
list-benchmarks, describe-benchmark Browse the built-in benchmark registry
validate-config Validate a config (--check-remote resolves configs and splits on the Hub)
build-index Build the n-gram index (.native + sidecars)
inspect-index Validate an index and list per-benchmark item and n-gram counts
scan Scan Parquet/JSONL shards against an index
run, run-hf Build or reuse a cached index, then scan local shards or an HF split
filter Scan, then write decontaminated copies of the shards with a manifest of the index used
selftest Plant benchmark items into clean text, scan it with your settings, and report recall
report Leak reports (JSON, CSV, Markdown) from a scan directory
inspect-run Validate a scan output directory
reproduce Re-run a scan from its _RUN_MANIFEST.json after checking input hashes
build-stopgrams-direct Learn stop-grams (frequent background n-grams) from a corpus
build-spaced-index Experimental: build the spaced-seed index

corpus-assay <command> --help shows every option. The full reference is in docs/cli.md.

Outputs

scan_out/
├── _FINAL_SUMMARY.json / .csv      # totals: docs scanned, contaminated (per channel), failed files
├── _RUN_MANIFEST.json              # scan config, index file and input hashes, benchmarks, tool versions
├── summaries/<shard>.summary.json  # per-shard counts (also the resume cache)
├── per_source_hits/<shard>.hits.bin
├── per_item_best/<shard>.json      # best single-document coverage of each item hit
├── contaminated_docs/<shard>.decontam.sample.ndjson   # first 500 flagged per shard (--all-records: every one)
└── report/                         # written by `report`
    ├── report.md / report.json
    ├── leak_by_benchmark.csv / .json
    ├── leak_by_shard.csv / .json
    ├── leak_by_item.csv / .json
    └── template_variants.csv / .json   # with --confirm: likely template variants

Re-running scan with the same inputs and config skips shards that are already done. File formats are documented in docs/outputs.md.

Filtering

filter takes the same options as scan, scans every shard (keeping every flagged record), and writes cleaned copies in the input format, at the inputs' relative paths, without touching the inputs:

corpus-assay filter --input data/ --index index.native --config config.yaml \
  --id-key id --out-dir data_clean/                  # drop rows with a flagged part
corpus-assay filter ... --granularity subdoc         # drop only the flagged parts

record granularity (the default) drops every row with a flagged sub-document; subdoc drops the flagged packed parts, empties flagged text fields and removes flagged chat turns, and drops a row only when nothing is left. Next to the data it writes _FILTER_MANIFEST.json (the index files' SHA-256, the benchmarks with their revisions and splits, the per-benchmark counts, gate and thresholds, tool versions, input and output shard hashes and row counts, and removals per benchmark), removed_ids.ndjson, a DATASET_CARD_SNIPPET.md to paste into the dataset card, and the scan itself under _scan/. It refuses a non-empty --out-dir unless --overwrite.

Checking recall

selftest measures what a scan with your settings would catch. It samples items from the index, plants them into synthetic filler (or copies of your first records, --host corpus --input ...), scans the planted file once per gate (and with the exact channel when --exact-channel is set), and prints recall per gate, channel and variant, the missed items, and false positives on unplanted hosts:

corpus-assay selftest --index index.native --config config.yaml --out-dir selftest/ \
  --n-items 50 --variants verbatim,case,whitespace,wrapper --exact-channel short

Items no enabled channel can match (fewer than min_hits n-grams and not covered by the exact channel) are listed apart and left out of recall. It exits non-zero when recall on matchable verbatim plants is below --min-recall (default 1.0), and writes planted.parquet and truth.jsonl so the result can be reproduced.

Reducing false positives

  • Reject patterns. The bundled normalization config drops n-grams that are multiple-choice scaffolding ("which of the following", "option a", ...). You can supply your own with corpus-assay --ngram-config my_config.json <command>. The override must be used for both build-index and scan; a mismatch is detected and refused.

  • Stop-grams. Build a list of n-grams that are frequent in a clean background corpus (for example Wikipedia), then pass it with --stopgrams. Matches on those n-grams are ignored:

    background:
      enabled: true
      corpus: {dataset: wikimedia/wikipedia, config_name: 20231101.en, split: train, fields: [text]}
    
    corpus-assay build-stopgrams-direct --config config.yaml --tau 0.001 \
      --out stopgrams.native --max-docs 1000000
    corpus-assay scan ... --stopgrams stopgrams.native
    
  • Item stop-grams. When the protected items repeat an instruction ("Write Python code to solve the problem..."), stopgrams.items: {max_item_share: 0.01} in the config (or build-index --item-stopgrams-share 0.01) drops the n-grams held by more than 1% of the items from the index.

  • Item gate. --gate-mode item flags a document only if a single benchmark item clears the thresholds on its own, rather than the union of all matches.

  • Template variants. --confirm reports, for each flagged pair, how many of the item's numbers the document has and its longest common substring with the question; report lists the pairs that look like other problems from the same template.

Python API

do_scan runs its workers with the spawn start method, which re-imports your main script in every worker. Call it under an if __name__ == "__main__": guard. Without the guard each worker would re-run the script's setup and the scan; do_scan detects this and raises a RuntimeError instead of hanging.

from pathlib import Path

from corpus_assay import build_index_hf, do_scan
from corpus_assay.config import (
    BuildIndexConfig,
    ScanConfig,
    load_decontamination_config,
)
from corpus_assay.scanner import OutputLayout, write_run_outputs

if __name__ == "__main__":  # required: scan workers are spawned processes
    cfg = load_decontamination_config("config.yaml")
    build_index_hf(
        BuildIndexConfig(
            out_path="bench.native",
            ngram=cfg.ngram,
            benchmarks=cfg.benchmarks,
            allow_unpinned=True,
        )
    )
    scan_cfg = ScanConfig(
        inputs=["data/"],
        text_keys=["text"],  # several columns: ["prompt", "chosen", "rejected"]
        id_key="id",
        index_path="bench.native",
        out_dir="scan_out",
        ngram=cfg.ngram,
        min_hits=cfg.min_hits,
        min_coverage=cfg.min_coverage,
    )
    run = do_scan(scan_cfg)
    write_run_outputs(
        layout=OutputLayout(Path(scan_cfg.out_dir)), run=run, cfg=scan_cfg
    )
    print(run.total_contaminated, run.contamination_rate)

Limitations

  • At most 64 benchmarks per index, because attribution is stored as a 64-bit mask. Use several indexes for more.
  • Text columns must be string columns (string, large_string, string view or string dictionary); chat columns must be lists of {role, content} structs.
  • JSONL shards are loaded fully into memory and converted to Parquet before scanning. Use Parquet for very large shards.
  • Documents are split on <|endoftext|>, and each part is scored separately. Records are named <id>-part-<k> (<id>/<field>[/<turn>]-part-<k> when several fields or a messages column are scanned). Pass --no-split to score each text whole.
  • Only verbatim overlap is detected; see above.

Documentation

Background

corpus-assay grew out of the decontamination pipeline used for the 13-gram contamination analysis of MixtureVitae (Appendix D). It keeps that pipeline's normalization, hashing and flagging thresholds, but some changes alter results, so default runs do not reproduce the paper's numbers exactly. docs/lineage.md lists the changes and how to get close to the original protocol.

Citing

If you use corpus-assay in your research, please cite it using CITATION.cff (GitHub's "Cite this repository" button). Every release is archived on Zenodo with its own DOI. The concept DOI 10.5281/zenodo.23125328 covers all versions.

Contributing

See CONTRIBUTING.md. Bug reports and pull requests are welcome.

License

Apache-2.0. See LICENSE.

Metadata

Release files for corpus-assay 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for corpus-assay 0.2.0
File Size Uploaded Provenance Actions
corpus_assay-0.2.0.tar.gz 471.2 kB With provenance Details for corpus_assay-0.2.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for corpus-assay 0.2.0
File Actions
corpus_assay-0.2.0-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for corpus_assay-0.2.0-cp312-abi3-win_amd64.whl
corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.12 abi3 Linux glibc 2.17+ x86-64 Details for corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.12 abi3 Linux glibc 2.17+ ARM64 Details for corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl
corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl CPython 3.12 abi3 macOS 10.12+ x86-64 Details for corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl

Total release size: 19.5 MB

Release files / corpus_assay-0.2.0.tar.gz

Download URL corpus_assay-0.2.0.tar.gz
Size 471.2 kB
Tags Source
SHA-256 checksum
How to use checksums
f1f771dba6a3d4fe9b78d56e757eb265737bc605668f4942df88ce6b9c8efc98
BLAKE2b-256 checksum
How to use checksums
0e47f0ab01615c7684ab24f9a228d2cceb1f09551dbfc1f5ecf6077f47d83adf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / corpus_assay-0.2.0-cp312-abi3-win_amd64.whl

Download URL corpus_assay-0.2.0-cp312-abi3-win_amd64.whl
Size 4.0 MB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
26445e16acdef6f968954cc795997a1225ec175b71cd00ca98a0f7f9ac184f50
BLAKE2b-256 checksum
How to use checksums
79e2c444ddcad2fd4d1aa43624cdd5a4459691a0b9feeda177fc4e12e92b1826
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL corpus_assay-0.2.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 4.0 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
1453cc592fac0c1fd7e8031f2c5ccf254f02832e9e3a4776665b4fdd018d2dab
BLAKE2b-256 checksum
How to use checksums
e842feaaec73b5b3088e629e0bcf3570f788a94092294cbe9c9394795ed8cf6c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL corpus_assay-0.2.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.7 MB
Tags CPython 3.12 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
3a9bebf94d2367e73367fa08861ab1494050894c45744baab2bac7616650eaad
BLAKE2b-256 checksum
How to use checksums
c3dd595604a2f140dfb8647c424e22bcb04a16984dcb2ef1e2597c8fe1a56619
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl

Download URL corpus_assay-0.2.0-cp312-abi3-macosx_11_0_arm64.whl
Size 3.5 MB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
184658fccc364597d4c09427e378a477f027976998c00fb420ad7add4082ec63
BLAKE2b-256 checksum
How to use checksums
845f424a1ece185f366de7ae8e77d37fb375064e8651d77b61ea7049c8bfab65
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Release files / corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl

Download URL corpus_assay-0.2.0-cp312-abi3-macosx_10_12_x86_64.whl
Size 3.8 MB
Tags CPython 3.12 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
8ed67b3974362f446bacaa71ad052cfc7919fe5e01ce3a8486ab81f8a757b382
BLAKE2b-256 checksum
How to use checksums
0f5c4a0d112ce55d797b1128b00b71f7c4ba2c0296bc0a1849331a115b01a16f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

Security for corpus-assay 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for corpus_assay-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 7, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

6 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page