toolkit-mmqa: dataset pre-flight and contamination check

A pip-installable, zero-GPU QA gate for fine-tuning and evaluation datasets. Run it before you train or evaluate; it fails CI when something is wrong and writes a report you can sign as an in-toto attestation.

• Contamination: find evaluation and benchmark records that also appear in the training data (exact, word 13-gram and MinHash matching), with every leaked record and the training records it matches.
• Record-level dedup: exact and near-duplicate records in JSONL, JSON, CSV or text datasets (or Hugging Face Hub splits), with a deduplicated copy.
• File-level QA: byte-identical files, near-duplicate text files, a per-file SHA-256 manifest, and file-by-file diffs between dataset versions.
• Media checks: corrupt images, resolution statistics and perceptual-hash near-duplicates; corrupt, truncated or empty audio and durations.
• Reports: every command writes one canonical-JSON report envelope (an in-toto Statement v1) with a pass / fail / error verdict that matches the exit code.

The core has no dependencies. Heavy integrations are optional extras. Video is not supported yet (video files are compared as bytes only). See Capability status.

5-minute example: is GSM8K's test set in its training set?

GSM8K (grade-school math, MIT license) ships a 7,473-question training split and a 1,319-question test split.

pip install "toolkit-mmqa[fast] @ git+https://github.com/AKIVA-AI/toolkit-mmqa.git"

curl -L -o gsm8k_train.jsonl https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/train.jsonl
curl -L -o gsm8k_test.jsonl https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl

toolkit-mmqa contamination --train gsm8k_train.jsonl --eval test=gsm8k_test.jsonl \
  --field question --format markdown
echo "exit code $?"

Output (0.4 s):

Verdict: **fail** (exit code 1)

| Target | Records | Contaminated | Fraction |
|---|---|---|---|
| test | 1319 | 3 | 0.0023 |

| Target | Index | Methods | Preview |
|---|---|---|---|
| test | 581 | ngram | Max plans to watch two movies this weekend. The first movie is 1 hour and 30 min |
| test | 602 | ngram | A plane travels 1200 miles in 3 hours. At the same rate, how many additional hou |
| test | 632 | ngram | Max bought stamps at the post office. Some of the stamps had a snowflake design, |
exit code 1

Three test questions share at least one 13-word sequence with a training question. The JSON report (--out contamination.json) says exactly where; for test question 602, 7 of its 13 distinct 13-grams occur in training questions 1314 and 5162:

{
  "index": 602,
  "methods": ["ngram"],
  "ngram": {"matched": 7, "overlap": 0.538462, "size": 13, "total": 13},
  "preview": "A plane travels 1200 miles in 3 hours. At the same rate, how many additional hours would it take to travel an additional 2000 miles?",
  "target": "test",
  "train_match_count": 2,
  "train_matches": [
    {"id": null, "index": 1314, "source": "gsm8k_train.jsonl"},
    {"id": null, "index": 5162, "source": "gsm8k_train.jsonl"}
  ]
}

The training split itself has no exact duplicate questions but 7 near-duplicate pairs (for example, questions 1174 and 7233 at estimated similarity 0.91). Finding them takes 2.8 s:

toolkit-mmqa dedup --input gsm8k_train.jsonl --field question --near-dup \
  --write-deduped gsm8k_train.dedup.jsonl --out dedup.json

To make this a CI gate, keep the exit code: 0 pass, 1 fail, 2 error. Allow a small rate with --max-contamination 0.01, or add --methods exact,ngram,minhash to also catch lightly edited copies. To check public benchmarks such as MMLU or HumanEval, see docs/benchmarks.md.

GitHub Action

- uses: actions/checkout@v4
- uses: AKIVA-AI/toolkit-mmqa@v1.0.0
  with:
    args: contamination --train data/train.jsonl --eval data/test.jsonl --field question
    extras: fast            # optional: fast, image, audio, hf, signing
    report: mmqa-report.json

The step fails when the verdict is fail or error, writes the Markdown summary to the job summary, and exposes verdict and report outputs. Pass any command's arguments without --out. The tag exists once v1.0.0 is released; until then use a commit SHA.

Capability status

Install

Not on PyPI yet. Install from source:

git clone https://github.com/AKIVA-AI/toolkit-mmqa.git
cd toolkit-mmqa
pip install .              # core, no dependencies
pip install ".[signing]"   # adds `cryptography` for --sign / verify
pip install ".[image]"     # adds Pillow for --image-checks
pip install ".[audio]"     # adds soundfile for FLAC / Ogg / MP3 / AIFF in --audio-checks
pip install ".[fast]"      # adds NumPy: vectorized MinHash (same results, ~12x faster)
pip install ".[hf]"        # adds datasets: read Hugging Face Hub datasets (hf:...)

Requires Python 3.10+.

Quick start

# Exact duplicates + per-file manifest
toolkit-mmqa scan --root ./my-dataset --out scan.json

# Also look for near-duplicate text files
toolkit-mmqa scan --root ./my-dataset --near-dup-text --near-dup-threshold 0.85 --out scan.json

# Summary statistics
toolkit-mmqa report --input scan.json

# What changed between two versions of a dataset?
toolkit-mmqa diff --old scan_v1.json --new scan_v2.json

# Did any test question leak into the training set? (exit 1 if so)
toolkit-mmqa contamination --train train.jsonl --eval test.jsonl --field question --out contamination.json

# Duplicate and near-duplicate records in a JSONL file, plus a clean copy
toolkit-mmqa dedup --input train.jsonl --near-dup --write-deduped train.dedup.jsonl --out dedup.json

# CI gate: exit 1 if any file is duplicated
toolkit-mmqa scan --root ./my-dataset --fail-on exact-duplicates --out scan.json

CLI reference

Global flags

Exit codes (the same for every command):

Output: --out and --format

Every command that writes a report takes:

scan

Recursively hashes every file under --root and groups byte-identical files.

Near-duplicate text. With --near-dup-text, every hashed file that is UTF-8 text (no NUL byte, strict decode) is shingled into character trigrams and compared with 128-permutation MinHash. The similarity is an estimate of the Jaccard similarity of the trigram sets. Files that are not text are listed in skipped_non_text. Pairs are found with LSH banding. The band layout is chosen so that every pair whose estimate reaches the threshold is always compared, so LSH gives the same result as comparing all pairs. All text files are loaded into memory; use --extensions and --max-file-size to bound large datasets. Exact duplicates also appear as near-duplicates (similarity 1.0).

Image checks. With --image-checks, every file with an image extension (jpg, jpeg, png, gif, bmp, webp, tif, tiff, ppm, pgm, pbm) is fully decoded with Pillow; files that fail (wrong format, truncated data, decompression-bomb limits) are listed as corrupt with the error. Each decoded image gets a 64-bit pHash and dHash with the same definitions as the imagehash library (grayscale, Lanczos resize, DCT or neighbour differences). Images whose pHashes differ in at most --image-hash-distance bits are grouped as near-duplicates; resized and re-compressed copies usually differ by 0 to 4 bits, unrelated images by about 32. Candidate pairs are found by splitting hashes into distance + 1 chunks, which cannot miss a pair within the distance.

Audio checks. With --audio-checks, every file with an audio extension (wav, flac, ogg, oga, opus, mp3, aif, aiff) is decoded to the end. PCM WAV uses the standard library, which also compares the frame count in the header with the data actually present, so truncated files are caught. Other formats, and WAV encodings the standard library cannot read, use libsndfile through the [audio] extra; without it they are listed as unsupported, not corrupt. Files with no audio frames count as corrupt. Duration is frames divided by the sample rate.

report

diff

Files are matched by relative path. Both scans must contain the files manifest (scans made by versions before the manifest was added are rejected with exit code 2; re-scan to diff them).

contamination

Finds evaluation or benchmark records that also appear in the training data. Use it for train/eval split overlap (--eval) and for benchmark contamination (--benchmark); both are checked the same way.

toolkit-mmqa contamination --train train.jsonl --eval test=test.jsonl \
  --field question --out contamination.json

Input files are JSONL, a JSON array, CSV (with a header) or plain text (one record per line), chosen by extension. A source can also be a Hugging Face Hub dataset split, hf:NAME[:CONFIG]:SPLIT[@REVISION] (for example hf:openai/gsm8k:main:test), with the [hf] extra; it is streamed. In the report, a Hub source has uri hf://datasets/NAME, and its digest is the SHA-256 of the records read (one canonical JSON line {"id", "index", "text"} per record), so pin @REVISION for a reproducible digest. Reading is fail-closed: a malformed line or a missing field is an error (exit 2) that names the file and line, and with --out an error report is written.

Methods. Text is normalized first (Unicode NFKC, lowercase, \w+ tokens), so case, punctuation and spacing do not hide a copy.

• exact: the normalized texts are equal.
• ngram: the record shares a word 13-gram with a training record. This is the definition from the GPT-3 contamination study (Brown et al. 2020, Appendix C). The report gives the fraction of the record's distinct n-grams found in training data. Records of 8 to 12 tokens are flagged when the whole record appears in a training record; shorter records are checked by exact only and counted in ngram_unchecked_count.
• minhash: estimated Jaccard similarity of character-trigram sets is at least the threshold. It catches light edits. The LSH band layout never misses a pair whose estimate reaches the threshold.

The evaluation side is indexed in memory and training files are streamed, so memory grows with the evaluation set, not the training set.

dedup

Finds duplicate records inside text datasets (not just duplicate files).

toolkit-mmqa dedup --input train.jsonl --field text --near-dup \
  --write-deduped train.dedup.jsonl --out dedup.json

Two records are exact duplicates when their normalized texts are equal (NFKC, lowercase, \w+ tokens). Duplicates are grouped transitively (exact and near groups are merged) and the first record of each group, in input order, is kept; the others are listed as removable. Records with no word characters are counted as empty_record_count and never grouped. All records are held in memory.

verify

Prints Signature is VALID (exit 0) or Signature is INVALID (exit 2). A detached signature covers the exact bytes of the report file, including the per-file manifest and any near-duplicate results, so a valid signature means the report is unchanged since signing. If there is no .sig file, verify checks the embedded signature field of a --format legacy-json scan. It does not check that files on disk still match; re-scan and compare the subject digest for that.

Create a key pair with the Python API:

from pathlib import Path
from toolkit_mmqa.signing import generate_ed25519_keypair

kp = generate_ed25519_keypair()
Path("mmqa_private.pem").write_text(kp.private_key_pem)  # keep this private
Path("mmqa_public.pem").write_text(kp.public_key_pem)

The private key is written unencrypted; protect it with filesystem permissions.

Any report can also be signed with the toolkit's shared signing tool, the optional toolkit-ml-provenance (Ed25519, or Sigstore keyless with its sigstore extra):

toolkit-mlsbom sign-file scan.json
toolkit-mlsbom verify-file scan.json

Report envelope

By default every report is an in-toto Statement v1, the attestation format used by SLSA and Sigstore, written as canonical JSON (UTF-8, sorted keys, no whitespace, trailing newline) so its SHA-256 is stable. The format is specified in docs/report-envelope.md with a JSON Schema in schemas/report-envelope.v1.json.

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{"name": "my-dataset", "digest": {"sha256": "5d1c..."}}],
  "predicateType": "https://github.com/AKIVA-AI/toolkit-mmqa/report/v1",
  "predicate": {
    "tool": {"name": "toolkit-mmqa", "version": "1.0.0"},
    "kind": "mmqa.scan",
    "created_at": "2026-09-26T18:00:00Z",
    "verdict": "pass",
    "exit_code": 0,
    "inputs": [],
    "summary": {"file_count": 3, "duplicate_group_count": 1, "failed_checks": [], "...": "..."},
    "details": {"files": {"...": "..."}, "duplicates": [["a.txt", "b.txt"]], "...": "..."}
  }
}

Dataset digest. For scan, the subject is the scanned directory and its digest is the SHA-256 of the canonical JSON (sorted keys, no whitespace, no trailing newline) of {relative_path: sha256} over every hashed file. Paths use / on every platform. Two scans of identical contents have the same subject digest, so the report attests to exactly which files were checked.

predicate.summary by kind

For report and diff the subject is the (newer) scan's dataset and inputs lists the scan files read, with their SHA-256.

predicate.details for mmqa.scan

With --format legacy-json the scan prints this details object on its own (plus an embedded signature with --sign).

predicate.details for mmqa.contamination

The subject lists the evaluation and benchmark files; inputs lists the training files, each with its SHA-256.

predicate.details for mmqa.dedup

predicate.details for mmqa.diff

A file that was renamed shows up as one removed path and one added path.

mmqa.report summary fields

Performance

MinHash signatures (used by --near-dup-text, dedup --near-dup and the minhash contamination method) are computed in pure Python by default. With the [fast] extra they are vectorized with NumPy and are bit-identical (tested). Measured with benchmarks/bench_minhash.py in a container limited to 2 CPUs (Python 3.12, 128 permutations, 2,000 documents of about 870 characters):

End to end, contamination --methods exact,ngram,minhash on GSM8K (7,473 training questions against 1,319 test questions) takes 30.9 s in pure Python and 2.7 s with [fast]; exact,ngram alone takes 0.4 s.

scan --workers N hashes files on N threads. Everything is in memory and in one process: thousands to low millions of records, not web-scale corpora.

Python API

from pathlib import Path
from toolkit_mmqa import diff_scans, find_near_duplicates, generate_report, scan

result = scan(root=Path("./my-dataset"))
print(len(result.duplicates), "duplicate groups in", result.file_count, "files")

summary = generate_report(result.to_json())
print("distinct contents:", summary.unique_content_count)

texts = {"a.txt": Path("a.txt").read_text(), "b.txt": Path("b.txt").read_text()}
dedup = find_near_duplicates(texts, threshold=0.8)
print(dedup.near_duplicate_groups)

Contamination and record dedup:

from pathlib import Path
from toolkit_mmqa import Target, check_contamination, dedup_records, load_records
from toolkit_mmqa import SourceRecords

train = load_records(Path("gsm8k_train.jsonl"), fields=["question"])
test = load_records(Path("gsm8k_test.jsonl"), fields=["question"])
result = check_contamination([("train", train)], [Target("test", test)])
print(result.targets[0]["contaminated_count"], [r["index"] for r in result.records])

dups = dedup_records([SourceRecords("train", train)], near_duplicates=True)
print(len(dups.removable), "removable records")

Docker

The image includes the image, audio, fast and signing extras.

docker build -t toolkit-mmqa .
docker run --rm -v "$PWD:/data" toolkit-mmqa scan --root /data --image-checks --audio-checks

Development

pip install -e ".[dev]"    # all extras plus test tools (imagehash is the test reference)
pytest -q
ruff check src/ tests/
pyright src/

CI also runs the suite with no extras installed (pip install -e . pytest) to keep the core dependency-free; tests that need an extra skip themselves.

See CONTRIBUTING.md and CHANGELOG.md.

Contributing and security

Contributions are welcome: see CONTRIBUTING.md and the Code of Conduct. Please report security problems privately, as described in SECURITY.md.

Releasing

Releases are cut by pushing a vX.Y.Z tag. CI runs the tests, builds the sdist and wheel, checks them, attaches them to a GitHub Release and publishes them to PyPI with Trusted Publishing. RELEASING.md describes the process and how to verify a release.

License

Apache License 2.0. See LICENSE and NOTICE.

Releases before the relicensing remain available under the MIT License.