leakfence

Audits a train/test split for subject/session and overlapping-window leakage. Plain arrays, no framework.

Why

Overlapping sliding windows plus a naive shuffle-split gives inflated, irreproducible accuracy. Two flavors:

• Temporal: adjacent windows from one recording land on both sides. Not generalizing, recognizing a near-duplicate.
• Spatial: same subject (or subject+session) in both partitions. The model learns who, not the task.

MNE/Braindecode/MOABB/EEGDash give correct splitting primitives if you build on their objects. None of them audit a split you already have.

Install

pip install leakfence

numpy is the only runtime dependency. Plots need pip install leakfence[plot].

Use

from leakfence import audit_split

report = audit_split(
    train_idx=train_idx,
    test_idx=test_idx,
    subject=subject_ids,
    session=session_ids,          # needed for paradigm="cross-session"
    sample_range=sample_ranges,   # (start, end) per window, optional
    recording=recording_ids,      # scopes the temporal check
    paradigm="cross-subject",     # within-session | cross-session | cross-subject | None
)

print(report.summary())
report.raise_if_failed()          # or strict=True on audit_split()

paradigm matters: the same subject in both partitions is a bug for cross-subject, correct for cross-session. Pick wrong, get false positives.

One check runs unconditionally, without subject or sample_range: the same row index in both partitions, wrong under every paradigm. See report.index_overlap_count. Sampling with replacement makes shared indices intentional, so allow_index_reuse=True skips it.

Indices are normalized before anything is compared. Negative indices resolve against the dataset length, so -1 and n-1 are one row. Boolean masks expand to positions. Out-of-range, float, or wrong-length arguments raise. Length comes from subject/sample_range/recording; pass n_rows for negative indices with none of those.

Severity, not just pass/fail: report.subject_overlap_count, report.temporal_overlap_rate, report.max_window_overlap.

Is that severity surprising, or what chance gives you anyway?

from leakfence import permutation_test_group_overlap, permutation_test_temporal_overlap

permutation_test_group_overlap(train_idx, test_idx, subject_ids, n_permutations=999)
# PermutationResult(observed=10.0, null_mean=8.9, null_std=1.0, p_value=1.0, n_permutations=999)

Null: random train/test relabeling of the same items, same sizes. p_value is P(null <= observed), one-sided. A plain shuffle-split scores near 1, as leaky as chance because it is chance. A split-construction test, not a classifier-accuracy test (cf. Ojala & Garriga 2010).

Mind the direction. Small p_value here is good news, the opposite of "p < 0.05, therefore an effect is present". Writing "p = 0.03, confirming leakage" would be backwards. To avoid reporting the raw number:

result.beats_chance()          # True: better structured than a random split
result.percentile              # 97.0: beats 97% of random splits

.to_dict() carries p_value_meaning and beats_chance_at_0.05 for the same reason.

The statistic follows paradigm. Default "cross-subject" tests subject_overlap_count, low is good. "cross-session" tests session_pairs instead, since subject overlap there is expected and would score a correct split like a leaky one. "within-session" raises: no group-overlap invariant applies, use permutation_test_temporal_overlap.

Same data appearing twice anywhere, not just across train/test:

from leakfence import check_duplicates

groups, violations = check_duplicates(X, train_idx=train_idx, test_idx=test_idx)
# groups: {fingerprint: (row indices)} for every exact content match

SHA256 per row, the same primitive as guard_fit's test_fingerprints. A group straddling train/test is severity="error", an exact leak. A duplicate inside one partition is severity="warning": worth knowing about, not a split problem.

report.clean_test_idx() returns test_idx with contaminated windows dropped.

from leakfence.plot import plot_temporal_leakage (needs leakfence[plot]) draws per-window severity against time, one row per recording. Flat is clean, a spike shows where and how bad. Returns the axes; show=True also calls plt.show().

Not every overlap is equal:

from leakfence import pair_variance_ratio

pair_variance_ratio(stats.pairs, sample_range, recording, signal)

Overlap-region variance over recording baseline, per pair. Above 1 means an artifact or outlier, worse than a leak on plain background. Not a stats test.

Sample ranges are half-open, [start, end), so (0, 100) and (100, 200) do not overlap. If your epoching reports end as the last sample in the window, add 1 before passing ranges in.

See examples/quickstart.py.

Global-preprocessing leaks

Fitting a scaler/PCA/ICA on the whole dataset before splitting. Wrap the estimator, not the array:

from leakfence import guard_fit

scaler, violations = guard_fit(StandardScaler(), n_train=len(train_idx))
scaler.fit(X_full)             # 200 rows, 150 are train -> flagged
scaler.fit(X_full[train_idx])  # clean

A row-count check cannot catch the right number of rows drawn from the wrong ones: an off-by-one slice, a resample that swapped a row in and out. Fingerprint content instead:

from leakfence import fingerprint_rows, guard_fit

test_fps = fingerprint_rows(X_full[test_idx])
scaler, violations = guard_fit(StandardScaler(), test_fingerprints=test_fps)
scaler.fit(X_candidate)  # flagged if any row matches held-out content, whatever the row count

The two checks carry different confidence. n_train is a heuristic (severity="warning", a resampled train set could trip it innocently); test_fingerprints is a definitive content match (severity="error"). strict=True raises only on the error case. Works on anything with .fit(X, ...).

Fingerprints normalize integer width, -0.0/0.0 and NaN payloads. They do not normalize float precision: float32(0.1) and float64(0.1) are different numbers, so fingerprint after any cast your pipeline performs. Object and string dtypes raise instead of hashing, since ndarray(dtype=object).tobytes() serializes pointers and equal content would hash differently.

guard_fit patches the instance, which makes it unpicklable on purpose: silently dropping a check during a joblib round-trip would be worse. Give it a lifetime:

from leakfence import guarded, unguard

with guarded(scaler, n_train=len(train_idx)) as (scaler, violations):
    scaler.fit(X_train)
# restored on exit: picklable, cloneable, joblib-safe

scaler, violations = guard_fit(scaler, n_train=len(train_idx))
unguard(scaler)   # or undo it by hand

This catches your own mistakes, not evasion: type(est).fit(est, X), cloning and re-instantiation all bypass the patch.

Auditing a pipeline you didn't write

A collaborator's code, a paper's supplementary repo, something AutoML-generated, where the nested Pipeline/ColumnTransformer structure isn't already in your head. An unfitted pipeline has no rows and no train/test labels, so it cannot detect a leak, but it can say what is stateful and where, with no data:

from leakfence import lint_pipeline

lint_pipeline(pipe)
# [{'path': 'prep.num', 'stateful': True}, {'path': 'prep.pass', 'stateful': False}, {'path': 'clf', 'stateful': True}]

An inventory, not a detector. It cannot know whether pipe is the whole object passed to cross_val_score; that is a fact about the call site.

How much data do you actually have

Not a leak, not a model judgment. Window overlap inflates the nominal count, 75% overlap by about 4x:

from leakfence import independent_capacity

independent_capacity(sample_range, recording)
# {'sub-01': {'nominal_windows': 77, 'independent_capacity': 20, 'inflation_factor': 3.85,
#             'window_length': 384, 'window_length_varies': False}, ...}

independent_capacity counts how many windows would tile the recording without sharing a sample. It is an upper bound, not an estimate: EEG independence is set by autocorrelation time, and two adjacent non-overlapping windows of resting-state alpha are still correlated. Read it as "at most this many".

Real example: COG-BCI resting-state, 5 subjects, 75% overlap, 385 nominal windows, at most 100 independent. Same 385 validation/ reports.

Enough data for what, though. Covariance-based (Riemannian/SPD) pipelines estimate a channel by channel covariance: n(n+1)/2 free parameters, quadratic in channel count.

from leakfence import dimensionality_report

dimensionality_report(n_channels=14, sample_range=sample_range, recording=recording)
# {0: {'p': 105, 'window_length': 40, 'independent_capacity': 5, 'nominal_windows': 20,
#      'per_trial':     {'n_samples': 40, 'mp_ratio': 0.35, 'rank_deficient': False, ...},
#      'across_trials': {'n_samples': 5,  'mp_ratio': 2.8,  'rank_deficient': True, 'param_ratio': 21.0, ...}}}

Two questions live here, and conflating them is how you end up calling a full-rank matrix singular:

• per_trial: can one window's channel covariance be full rank at all? Channels vs. time samples inside the window. Marchenko-Pastur: mp_ratio = C/N, rank-deficient exactly when C > N. 14 channels from a 40-sample window is full rank.
• across_trials: enough independent windows for p = C(C+1)/2 parameters? param_ratio = p/N, a parameter-counting rule of thumb. param_ratio >= 1 (ill_conditioned) means shrink or regularize. Not a claim about rank.

across_trials uses independent_capacity, not nominal windows: overlap adds no independent covariance information either.

Paste-into-appendix report

Everything above, concatenated. No new numbers, just formatting:

from leakfence import experiment_report

print(experiment_report(report, sizes=sizes, dims=dims, dataset_name="toy resting-state", paradigm="cross-subject"))

Dataset: toy resting-state
Paradigm: cross-subject

Leakage audit:
  subject overlap: 0
  temporal overlap rate: 0.0%
  max window overlap: 0.0%
  verdict: no leakage detected

Effective sample size (upper bound: tiling, not autocorrelation):
  sub-01: 77 nominal, at most 20 independent (3.85x inflation)
  sub-02: 77 nominal, at most 20 independent (3.85x inflation)

Covariance estimability:
  sub-01: per-trial covariance full rank (14 ch / 384 samples, Marchenko-Pastur C/N=0.04)
       across trials under-determined (p=105 params / 20 independent windows, p/N=5.25)
  sub-02: per-trial covariance full rank (14 ch / 384 samples, Marchenko-Pastur C/N=0.04)
       across trials under-determined (p=105 params / 20 independent windows, p/N=5.25)

sizes and dims are optional; omit either and that section is skipped. These numbers come from running examples/quickstart.py's toy dataset, not from invention.

Same inputs, JSON instead of text, for logging runs or plotting the null distribution yourself:

from leakfence import experiment_report_dict

experiment_report_dict(report, dataset_name="toy resting-state", paradigm="cross-subject")
# {"dataset_name": "toy resting-state", "paradigm": "cross-subject",
#  "audit": {"ok": True, "subject_overlap_count": 0, ..., "group_stats": {...}, "temporal_stats": None},
#  "sizes": None, "dims": None}

report.to_dict() works the same way, capping the temporal pair list at 1000 entries; pass max_pairs=None for all of it. PermutationResult.to_dict() includes the full null_distribution. GroupOverlapStats, TemporalOverlapStats and Violation all have to_dict(). JSON object keys are always strings, so an int recording id round-trips as "0".

Building a split instead of auditing one

audit_split checks a split you already have. split_windowed constructs one that is leak-free by construction, Roth (2026)-style (arXiv:2603.10742), adapted for EEG since Roth's split_group/split_temporal don't know windows exist:

from leakfence import split_windowed, assess_once

part = split_windowed(subject, session=session_ids, paradigm="cross-session", test_frac=0.2)
part.train_idx, part.test_idx     # whole (subject, session) groups, never split
assess_once(part)                 # raises LeakageError on a second assessment

cross-subject and cross-session assign whole groups to one side, which rules out temporal overlap for free. within-session cuts each recording chronologically and drops any train window straddling the boundary. assess_once is process-scoped: it resets on restart, the same limitation as Roth's version.

reset_assessments() forgets every partition assessed so far, so re-running a notebook cell is not indistinguishable from reusing a test set. Calling it in a script defeats the point.

Covers the three MOABB paradigms only, not general-purpose splitting.

Not doing (yet)

• General-purpose split construction. For anything outside the three paradigms, use Braindecode/MOABB.
• No CLI.

Adjacent tools

• timefence: temporal leak in tabular feature/label joins. No windows.
• DataSAIL: leakage-reduced splits for biomolecular similarity data. Not time-series.
• EEGDash / Braindecode: correct splitters, but only on their own data objects.

Citation

split_windowed and assess_once adapt the split-then-assess-once discipline from:

Roth, S. (2026). A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time. arXiv:2603.10742.

License

MIT