contamination-audit
Generate item-level signals that help reviewers investigate possible evaluation-data leakage.
contamination-audit is for benchmark maintainers and evaluation engineers who want a local, inspectable preflight before deeper contamination analysis. It combines n-gram overlap, answer-pattern checks, canary matching, synthetic registry-hash matching, lexical similarity, and an optional sentence-transformers backend.
Inspectable Output
The CLI writes JSON with item_count, finding_count, a bounded aggregate risk score, and every detector-level finding. Findings include item and reference ids, detector name, matched corpus marker, canary, or similarity score where applicable.
The Python run API accepts separate items, references, outputs, and registry data. The CLI currently accepts only --eval-data; it uses those same rows as its comparison references and does not accept model outputs, so it is a smoke-test path rather than a held-out-corpus audit.
Runtime Boundary
The default lexical path runs locally with no network dependency. The built-in pile, c4, and hf-mmlu registry entries are synthetic marker hashes; the package does not download or search those corpora.
The optional sentence-transformers backend loads the named model locally. Its underlying library may download model files when they are not already cached, so use a local model/cache when runtime network access is not allowed.
Install
python -m pip install contamination-audit==0.1.2
Optional semantic embeddings:
python -m pip install "contamination-audit[embedding]==0.1.2"
For development from a clone:
python -m pip install -e .
Quickstart
From a repository checkout, run the no-download path:
contamination-audit run \
--eval-data examples/eval.jsonl \
--skip-embedding \
> contamination-report.json
See docs/methodology.md for detector and data-boundary details.
Release Status
Registry status verified July 13, 2026: version 0.1.2 is published on PyPI and tagged v0.1.2 in the public repository. The project is alpha software. No claim of contamination detection coverage, clean data, or adoption is made.
Limits
This package emits diagnostic signals, not proof that data is contaminated or clean. The aggregate risk score is a simple finding-count heuristic, the bundled corpus registry is synthetic, and reviewers must interpret evidence in context.
Next Action
Run the no-download CLI path to inspect the report shape, then call the Python API with approved, separated references and model outputs before using any finding in a contamination review.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file contamination_audit-0.1.2.tar.gz.
File metadata
- Download URL: contamination_audit-0.1.2.tar.gz
- Upload date:
- Size: 9.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b12fb409602e7e0fd304a37d2d0a1785a055e89f7c9ea7f2e5595fa52a9daf0d
|
|
| MD5 |
f8667c59ef1160ee7b7fff1593a2b591
|
|
| BLAKE2b-256 |
75865527815200de255b2ccee2c3eb743b9a6619943cd1618521d50ad4868395
|
Provenance
The following attestation bundles were made for contamination_audit-0.1.2.tar.gz:
Publisher:
release-python.yml on auraoneai/contamination-audit
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
contamination_audit-0.1.2.tar.gz -
Subject digest:
b12fb409602e7e0fd304a37d2d0a1785a055e89f7c9ea7f2e5595fa52a9daf0d - Sigstore transparency entry: 2165375849
- Sigstore integration time:
-
Permalink:
auraoneai/contamination-audit@743f1774583c79419eca6aa86178db09950c49b1 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/auraoneai
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-python.yml@743f1774583c79419eca6aa86178db09950c49b1 -
Trigger Event:
push
-
Statement type:
File details
Details for the file contamination_audit-0.1.2-py3-none-any.whl.
File metadata
- Download URL: contamination_audit-0.1.2-py3-none-any.whl
- Upload date:
- Size: 9.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
00d5d63f3f7a6b8fb8fb1bc01784f60a1bc7921cbacbb7a09c95d227948397f0
|
|
| MD5 |
1b1eb379f6b28e104d6f25f7d6aae7af
|
|
| BLAKE2b-256 |
436ddc6ee1600a2bf161b47b8c0203369224996d53ca219dd09814510ac334ca
|
Provenance
The following attestation bundles were made for contamination_audit-0.1.2-py3-none-any.whl:
Publisher:
release-python.yml on auraoneai/contamination-audit
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
contamination_audit-0.1.2-py3-none-any.whl -
Subject digest:
00d5d63f3f7a6b8fb8fb1bc01784f60a1bc7921cbacbb7a09c95d227948397f0 - Sigstore transparency entry: 2165375852
- Sigstore integration time:
-
Permalink:
auraoneai/contamination-audit@743f1774583c79419eca6aa86178db09950c49b1 -
Branch / Tag:
refs/tags/v0.1.2 - Owner: https://github.com/auraoneai
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release-python.yml@743f1774583c79419eca6aa86178db09950c49b1 -
Trigger Event:
push
-
Statement type:
