targetleak

Did your model learn, or did you leak?

targetleak reads a training set and tells you which columns are secretly carrying the answer — before you ship a model whose test score was never real.

pip install targetleak
targetleak --demo

For .parquet files: pip install "targetleak[parquet]"

CRITICAL target-proxy [cancellation_reason]
         alone separates the target at 1.0000 (true AUC 1.0000), 29.9 SE
         above chance. One column should not nearly solve the target - this
         is very likely computed from the answer, or recorded after it was
         known.
         EVIDENCE: 'not_given': 0% positive (n=568) | 'moved': 100% positive
         (n=302) | 'price': 100% positive (n=330)
         FIX: Establish when this column receives its value. If it is
         written at or after the moment the target becomes known, it cannot
         be an input. Drop it, or rebuild it from data available strictly
         before the prediction cutoff.

VERDICT: 3 critical leak(s). Do not trust this model's test score until they
are resolved.

Every finding carries the measurement it rests on, how far that stands from chance, and what to do about it.

The idea

A model needs many features to reach AUC 0.95. A leak gets there with one.

So any lone column that nearly solves the target is the prime suspect — it is usually a proxy for the answer that will not exist at prediction time. That single check finds most real leaks, and it needs no model, no training run, and no labels beyond the ones you already have.

Every finding comes with what to do about it. Naming a leak without saying how to fix it is a scolding, not a tool.

Usage

targetleak data.csv  --target churned
targetleak data.parquet --target y --split is_test --group user_id
targetleak data.csv  --target y --json          # for CI
targetleak data.csv  --target y --no-fixes      # findings only

As a library:

import targetleak

findings = targetleak.analyse(df, target="churned")
print(targetleak.report(findings))

if any(f.severity == "critical" for f in findings):
    raise SystemExit("refusing to train on a leaking dataset")

Exit code is 1 when anything critical is found, so this works as a gate:

targetleak data.csv --target y && python train.py

In CI

- run: pip install targetleak
- run: targetleak data/train.csv --target churned

A checker with no way to accept a finding can never go green, so it gets deleted. Reviewed a column and decided it is fine? Name it:

targetleak data.csv --target y --ignore customer_tier,promo_code

Ignored columns are still listed, at info level. A silent suppression is how the next real regression gets missed.

There is a ready-made workflow in .github/workflows/leak-check.yml that comments the findings on the pull request and attaches the HTML report. Your data never leaves your runner; only the findings reach the comment.

What it checks

Binary, multiclass (one-vs-rest) and continuous targets. CSV, TSV, Parquet.

Every score is also required to stand clear of the null by several standard errors, with the bar rising as more columns are tested. Without that, 60 columns of pure noise on 20 rows produce a "critical leak" - the threshold alone has no idea how much data it is looking at.

For a raw column that null is analytic (Hanley-McNeil). For a target-encoded one it is measured, by permuting the target and re-encoding, because an encoding built from the target is not independent of it - assuming otherwise made a four-category noise column clear the bar 3.7% of the time against a nominal 0.05%. The permutations run only for columns that already cleared the score threshold, so the cost is per candidate, not per column.

Two details that make it work

Categoricals are scored with out-of-fold target encoding. Encode in-fold and every high-cardinality column scores ~1.0, because each category predicts its own mean — the tool would flag every legitimate feature and be useless. A 120-city column of pure noise scores 0.50 here, as it should.

Names are checked as well as statistics. A 5-day forward label is future information whether or not it correlates well with the label you are training on. No AUC threshold can catch that; the column's name gives it away. Found on real data where seven label_* columns sat in the feature matrix scoring only 0.60–0.76 — invisible to statistics, obvious from their names.

Benchmark

Synthetic tests where the same author writes both the leak and the detector prove nothing. This runs against data nobody here constructed:

python benchmark/run_benchmark.py

Recall on documented leaks: 2/2. False positives on clean data: 5 across 121 columns in 5 datasets.

Titanic's boat and body are the textbook leakage example — a lifeboat number exists only for people who got into a lifeboat, a body-recovery number only for people who did not survive. The other five ship with scikit-learn and are among the most-studied datasets in the field; if they leaked, it would be famous. Every critical finding there is counted against the tool.

The five false positives are real and are not going to be tuned away. On iris, petal length gives AUC 1.0 against setosa. That is identical in every measurable respect to a leak — the difference is that iris is genuinely an easy problem, and no statistic can see the difference. It is the tool's central limitation, so the benchmark counts it as a failure rather than explaining it away, and a critical finding is worded to name both possibilities.

The benchmark also earned its place immediately: it caught a miss on body. That column identifies only 121 of 1,309 passengers, so its missingness AUC is 0.575 and the ranking check walked straight past it — even though every one of those 121 died. A column can give the answer away on a subset of rows while looking like noise overall, and that check now exists because a dataset we did not write exposed its absence.

A real find

One dataset is not a validation set, and this one is the author's own project rather than an independent trial - so read it as a worked example, not a benchmark. It is here because the bugs were real and nobody had planted them.

Run against a 448,000-row × 69-column production training set for a live trading system, targetleak reported that the target column was 79% null and that 17 features were constant on the rows that actually carried labels — including an entire insider-trading feature family fed by its own weekly ingestion pipeline.

Those features varied normally across the file: 72,668 distinct values for one of them. But of the 72,828 rows where it was non-zero, zero carried a label. The label column and the enriched features covered disjoint time periods. The model had never seen a single non-zero value from that pipeline, and its owner had been investigating why the model showed no edge.

That is the class of bug this finds: not a mistake in the modelling, a mistake in what reached the model.

What it does not do

• It does not detect preprocessing leakage (a scaler fit before the split). That lives in your code, not your data — read your pipeline.
• A suspiciously-predictive finding is not proof. Some features really are that good. It tells you where to look.
• A discrete predictor against a continuous target is scored by how well the target separates each of its groups, not by correlation. |Spearman| for a two-group predictor is capped at 0.866 at an even split and collapses from there, so a flag identifying the top 2% of a revenue target used to score 0.62 and read as noise.
• The power gate is a Bonferroni-flavoured approximation, not an exact test. It is there to stop small samples producing confident nonsense, and it will occasionally hold back a real finding on a small dataset - which is reported as underpowered rather than hidden.
• It cannot know your business. A column that is legitimate at prediction time in one system is a leak in another. You decide; it points.

Development

git clone https://github.com/ShriyansBharuka/TargetLeak
cd TargetLeak
pip install -e ".[dev]"
pytest

License

Apache-2.0.