leakcheck

Catch data leakage before it ruins your model.

Machine-learning data leakage, not security leaks: leakcheck finds problems in train/test splits that make a model look better than it really is.

A model that scores 99% in testing and fails in production usually has a leak: test rows that were also in training, the same patient on both sides of the split, a feature that secretly contains the answer, or a time series split in the wrong order. leakcheck is a linter for your train/test split. Run it before you train.

$ leakcheck train.csv test.csv --target price_nok

leakcheck  train: 1,600 rows × 7 cols   test: 500 rows × 7 cols

  ✗ 60 test rows (12.0%) also appear in train
  ✗ 62 test rows (12.4%) are near-duplicates of train rows (≥75% of values match) — about 30 more than the 32 expected by chance
  ✗ Column 'final_price_eur' has correlation 1.000 with target 'price_nok'
  ⚠ Column 'listing_id' looks like an ID (unique per row) — drop it before training
  ⚠ 100.0% of test rows on 'listed_date' are not after the latest train date (2024-05-14)

3 problem(s), 2 warning(s) — likely leakage

Install

pip install leakcheck-ml

The package is leakcheck-ml on PyPI; the command and the import are both leakcheck.

Usage

leakcheck train.csv test.csv --target label             # basic
leakcheck train.parquet test.parquet -t label           # parquet (pip install "leakcheck[parquet]")
leakcheck train.csv test.csv -t label --time-col date   # enforce a time-based split
leakcheck train.csv test.csv -t label -g patient_id     # enforce a group split (no patient on both sides)
leakcheck train.csv test.csv -t label --similarity 0.9 # stricter near-duplicate matching
leakcheck train.csv test.csv -t label --json            # machine-readable output
leakcheck train.csv test.csv -t label --strict          # fail on warnings too

The exit code is 1 when problems are found, so you can drop it into CI and block a pipeline that would train on leaky data.

What it checks

Try it

python examples/make_demo.py       # two demo datasets with planted leaks
leakcheck examples/train.csv examples/test.csv --target price_nok
leakcheck examples/patients_train.csv examples/patients_test.csv --target diagnosis

  ✗ 176 of 183 'patient_id' groups (263 test rows, 93.9%) also appear in train — split by group instead of by row

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

Roadmap

• Near-duplicate detection with MinHash + LSH (rows that are almost identical)
• Group leakage (same patient/user in both train and test)
• GitHub Action
• Publish to PyPI
• HTML report

License

MIT