auditeeg

Audits for EEG decoding benchmarks: how much of a reported score comes from the evaluation rather than from the decoder.

Each audit compares a leaky evaluation with its clean twin on the same participants and reports the inflation, how many participants show it, a one-sided paired Wilcoxon test, a rank-biserial effect size and Holm correction across a family of audits.

Status: early release. Version 0.0.1 contains the paired statistics (auditeeg.stats). The protocol audits (segment leakage, supervised spatial filters, transductive alignment, test-set selection of preprocessing, epochs and seeds), the toolchain checks and the reporting audits will be released with the accompanying paper.

Install

pip install auditeeg

Example

import numpy as np
from auditeeg import audit_leakage_mechanism, audit_family

clean = np.array([0.62, 0.55, 0.71, 0.48, 0.66, 0.59, 0.73, 0.52, 0.68])  # one score per participant
leaky = clean + np.array([0.05, 0.09, 0.02, 0.11, 0.04, 0.07, 0.03, 0.08, 0.06])

audits = [audit_leakage_mechanism(leaky, clean, name="segment leakage")]
audit_family(audits)  # adds Holm-adjusted p-values
print(audits[0]["inflation_margin"], audits[0]["subject_consistency"], audits[0]["p_holm"])

Links

• Project page: https://auditeeg.github.io/

License

BSD-3-Clause.