noleak

Content-addressed dataset IDs and train/eval leakage checks for AI/ML.

Published numbers are only as honest as the split behind them. noleak gives every corpus a stable fingerprint (like a Docker image ID) and measures how much of your eval set already appeared in train — exact copies, GPT-3-style word n-grams, and MinHash near-duplicates.

Zero runtime dependencies. Stdlib only.

from noleak import check, fingerprint

train = ["the model was trained on Wikipedia dumps and licensed books"]
eval_set = ["The model was trained on Wikipedia dumps and licensed books"]

print(fingerprint(eval_set))
# noleak-fp-v1:a1b2c3d4e5f6

report = check(train, eval_set)
print(report)
# exact 1/1 (100.00%)  status FAIL

Why this library

Influential AIML libraries are usually one primitive done well (einops, safetensors, tqdm). The primitive here:

1. Identity — an order-independent hash of unique normalized examples, so a paper can write eval id noleak-fp-v1:… and a reviewer can recompute it.
2. Leakage — exact overlap, 13-gram contamination (Brown et al., GPT-3), and character-shingle MinHash for paraphrases.

This does not replace scikit-learn Pipeline leakage tools such as splitguard (runtime fit() hooks). It answers a different question: did this text eval set leak from the train corpus?

Install

pip install noleak

Requires Python 3.10+. No extra packages.

For local development from this repo:

python3 -m pip install -e ".[dev]"

API

from noleak import check, exact_overlap, fingerprint, near_duplicates, ngram_overlap

fp = fingerprint(eval_texts)          # Fingerprint
report = check(train_texts, eval_texts)  # CheckReport
report.contaminated                   # bool (CI-friendly)
report.to_dict()                      # JSON-serializable

exact_overlap(train, eval_set)        # identical after normalize
ngram_overlap(train, eval_set, n=13)  # shared word n-grams
near_duplicates(train, threshold=0.8) # pairs inside one corpus

Records may be strings or dicts. Dicts use the first present of text, content, prompt, question, input, output.

CLI

noleak fingerprint eval.jsonl
noleak check --train train.jsonl --eval eval.jsonl
noleak check --train train.jsonl --eval eval.jsonl --json
echo $?   # 1 if contaminated, 0 if clean

JSONL, JSON lists, and plain text (one example per line) are supported. Use --field name when objects store the example under a custom key.

Fingerprint scheme (noleak-fp-v1)

1. Normalize each example (noleak-norm-v1): Unicode NFKC, strip, collapse whitespace, casefold.
2. SHA-256 the UTF-8 bytes of that string.
3. Sort unique example hashes.
4. SHA-256 noleak-fp-v1\n plus those hashes, one per line.

Row order and duplicate rows do not change the digest; n_examples vs n_unique still records multiplicity.

Leakage methods

Short eval lines (fewer than n tokens) are hashed as a single gram so they are not skipped.

What this will not do

• Catch target leakage (a feature derived from the label).
• Catch preprocessing fitted on the full matrix (use a Pipeline or a runtime hook).
• Prove a model never saw an eval item during pretraining of a closed corpus you cannot hash.
• Replace embedding-based semantic dedup (sentence-transformers). MinHash is deterministic and model-free; that is the point.

Development

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev]"
pytest
python -m build

License

MIT