SciTeX Seizure Metrics (scitex-seizure-metrics)

Unified evaluation library for seizure detection and forecasting — sample-based, alarm-based, and the bridge between them.

Full Documentation · uv pip install scitex-seizure-metrics[all]

Problem and Solution

_{Table 1. Why seizure evaluation needs one library: scattered metrics, the untooled sample-vs-alarm collapse, and vague FP/hr denominators.}

Supported Metrics

Quick definitions for the metrics and policy knobs that recur throughout the README, the docstrings, and the cited papers.

The AlarmPolicy config knobs (SPH · SOP · cadence · refractory · alarm-threshold · FP-denominator) are documented inline on the dataclass and shown in the forecasting example below — they pin alarm-derivation, not metric definitions.

Quick Start

from scitex_seizure_metrics import detection, forecasting, AlarmPolicy

# Detection — per-window classification
rep = detection.evaluate(y_true, y_proba, threshold=0.5, fs=1)
print(rep.roc_auc, rep.pr_auc, rep.brier, rep.mcc)

# Forecasting — continuous stream with explicit alarm policy
policy = AlarmPolicy(
    sph_seconds=300, sop_seconds=600, cadence_seconds=60,
    refractory_seconds=600, alarm_threshold=0.5,
    fp_denominator="interictal",   # Mormann tradition
)
rep = forecasting.evaluate_stream(
    proba, times, seizures, policy,
    total_recording_time=24 * 3600,
)
print(rep.sensitivity, rep.fp_per_hour, rep.ioc, rep.time_in_warning_frac)

See examples/01_detection_quick_start.ipynb, examples/02_forecasting_quick_start.ipynb, and the other notebooks under examples/ for end-to-end workflows.

Forecasting classification metrics + lead time (v0.2.0)

The alarm regime reports the full confusion matrix and the warning time it actually delivered. TP = caught seizures, FN = uncaught, FP = alarms catching nothing; TN = interictal SOP-length "prediction opportunities" with no false alarm (n_opportunities = floor(interictal_seconds / SOP), TN = max(0, n_opportunities − FP) — the Snyder/Schelter/Mormann tradition; see docs/math/alarm_confusion_matrix.md and ADR-0001 for why "what is a true negative" is a documented convention).

import numpy as np
from scitex_seizure_metrics import AlarmPolicy, forecasting

# 3 seizures; 3 alarms, the first two of which catch a seizure
seizures = np.array([3600.0, 7200.0, 18000.0])
alarms   = np.array([3000.0, 6900.0, 12000.0])
policy = AlarmPolicy(sph_seconds=300, sop_seconds=600,
                     cadence_seconds=60, refractory_seconds=600,
                     fp_denominator="interictal")

rep = forecasting.evaluate(alarms, seizures, policy,
                           total_recording_time=24 * 3600, n_surrogate=50)

print(rep.n_tp, rep.n_fp, rep.extras["n_fn"], rep.n_tn, rep.n_opportunities)
# 2 1 1 135 136
print(round(rep.sensitivity, 3), round(rep.specificity, 3),
      round(rep.ppv, 3), round(rep.npv, 3), round(rep.forecasting_f1, 3))
# 0.667 0.993 0.667 0.993 0.667
print(rep.lead_time_mean, rep.extras["lead_times_seconds"])
# 450.0 [600.0, 300.0]   (delivered ≥ the 300 s SPH the policy required)

specificity / npv scale with the SOP-opportunity TN convention (read them with n_tn / n_opportunities); ppv / forecasting_f1 do not. Undefined ratios are NaN (fail-loud), never a silent 0. Observed lead time is distinct from the SPH constraint — SPH is the minimum required, lead time is what the system actually delivered (SPH ≤ lead ≤ SPH + SOP).

Demo

from scitex_seizure_metrics import detection, forecasting, AlarmPolicy

# Per-window detection metrics (sensitivity, false-positives/hour, ...)
m = detection.evaluate(y_true=labels, y_pred=preds, fs=256)
print(m.sensitivity, m.fp_per_hour)

# Forecasting metrics (Improvement-over-chance, AUROC, alarm count)
f = forecasting.evaluate(
    alarm_times=alarms, seizure_times=onsets,
    policy=AlarmPolicy(sph_seconds=300, sop_seconds=600,
                       cadence_seconds=60, refractory_seconds=600),
    total_recording_time=24 * 3600,
)
print(f.ioc, f.roc_auc)

graph LR
    Labels["per-window y_true / y_pred"] --> Det["detection.evaluate"]
    Onsets["seizure_times + alarm_times"] --> Fore["forecasting.evaluate"]
    Det --> Out["sensitivity / FP-per-hour / latency"]
    Fore --> Out2["IoC / AUROC / alarm count"]

_{Figure 1. Demo data flow: per-window labels feed detection, seizure/alarm times feed forecasting.}

Installation

uv pip install "scitex-seizure-metrics[all]"

Python ≥ 3.10 (CI tests 3.11 / 3.12 / 3.13). Through the SciTeX umbrella: pip install "scitex[seizure-metrics]" → scitex.seizure_metrics.*.

uv pip install scitex-seizure-metrics              # runtime only
uv pip install "scitex-seizure-metrics[plots]"     # + matplotlib for plots
uv pip install "scitex-seizure-metrics[all]"       # plots + docs + dev

Architecture

flowchart LR
    Probs["per-window proba<br/>+ ground truth"] --> Det["detection.evaluate"]
    Probs --> StreamIn["forecasting.evaluate_stream"]
    Policy["AlarmPolicy<br/>SPH · SOP · cadence · refractory · FP denom"] --> StreamIn
    Det --> RepDet["MetricsReport<br/>AUROC · AUPRC · Brier · MCC"]
    StreamIn --> RepFc["MetricsReport<br/>sensitivity · FP/hr · IoC · TIW"]
    RepDet -.->|"bridge analytic bounds"| RepFc
    RepFc --> Plots["plots: sensitivity vs FP/hr,<br/>IoC vs surrogate, cadence ablation"]

_{Figure 2. Architecture: detection and forecasting converge on MetricsReport; the bridge bounds the unreported regime.}

The split mirrors how the seizure-evaluation literature itself is organised — sample-based vs alarm-based vs the bridge — so a paper-faithful re-implementation lives in exactly one place. MetricsReport is the single object that travels between regimes; AlarmPolicy is the single object that pins every reproducibility decision an alarm-based metric requires.

6 Interfaces

from scitex_seizure_metrics import AlarmPolicy, forecasting

policy = AlarmPolicy(
    sph_seconds=300, sop_seconds=600, cadence_seconds=60,
    refractory_seconds=600, alarm_threshold=0.5,
    fp_denominator="interictal",
)
rep = forecasting.evaluate_stream(
    proba, times, seizures, policy,
    total_recording_time=24 * 3600, n_surrogate=1000,
)
print(rep.sensitivity, rep.fp_per_hour, rep.ioc, rep.time_in_warning_frac)
# Alarm-regime confusion metrics + observed lead time
print(rep.specificity, rep.ppv, rep.npv, rep.forecasting_f1)
print(rep.lead_time_mean, rep.lead_time_median,
      rep.extras["lead_times_seconds"])

# Operating curve across thresholds
df = forecasting.sweep_thresholds(proba, times, seizures, policy)

# Cadence ablation
policies = [AlarmPolicy(..., cadence_seconds=c) for c in [30, 60, 120, 300]]
df = forecasting.sweep_policies(proba, times, seizures, policies)

from scitex_seizure_metrics import detection
rep = detection.evaluate(y_true, y_proba, threshold=0.5, fs=1)
print(rep.roc_auc, rep.pr_auc, rep.brier, rep.mcc, rep.balanced_accuracy)

from scitex_seizure_metrics import bridge

bnd = bridge.sample_to_alarm(
    sample_sensitivity=0.79, sample_specificity=0.85,
    sop_seconds=600, cadence_seconds=60, refractory_seconds=600,
)
print(bnd.alarm_sensitivity_upper, bnd.fp_per_hour_upper)

from scitex_seizure_metrics import AlarmPolicy, plots, sensitivity_tiw

policy = AlarmPolicy(sph_seconds=0, sop_seconds=600,
                     cadence_seconds=60, refractory_seconds=600)

curve = sensitivity_tiw.sensitivity_tiw_curve(
    scores, policy, seizure_times=onsets, times=times, target_tiw=0.20,
)
print(curve.improvement_over_chance,        # AUC-like area above the diagonal
      curve.sensitivity_at_target_tiw,      # sensitivity at 20 % time-in-warning
      curve.tiw_at_target_sensitivity)      # time-in-warning at 75 % sensitivity

# Is the operating point above a time-matched coin?
sig = sensitivity_tiw.surrogate_above_chance(
    scores, policy, threshold=0.5, seizure_times=onsets, times=times,
)
print(sig.p_value, sig.ci_low, sig.ci_high)

plots.sensitivity_tiw([curve], save_path="fig_sens_tiw")  # png + pdf

from scitex_seizure_metrics.papers import andrade2024
out = andrade2024.metrics(
    y_true=labels, y_proba=preds,
    times_seconds=times, seizure_times=onsets,
)
print(out["sample_auroc"], out["alarm_sensitivity"], out["beats_chance_alarm"])
# Reproduces the side-by-side sample-vs-alarm panel from the paper.

from scitex_seizure_metrics import calibration, plots
cal = calibration.calibration_report(y_true, y_proba, n_bins=10)
print(cal.brier, cal.reliability, cal.resolution, cal.uncertainty,
      cal.expected_calibration_error)
plots.reliability_diagram(cal)

from scitex_seizure_metrics import plots
plots.sensitivity_vs_fp_per_hour(sweep_df)        # operating curve
plots.sensitivity_tiw([curve])                    # sensitivity vs time-in-warning (Karoly 2017 Fig 6)
plots.ioc_vs_surrogate(sweep_df)                  # model vs chance
plots.cadence_ablation(policy_sweep_df)           # FP/hr vs cadence
plots.sample_vs_alarm_scatter(per_patient_df)     # the Andrade 2024 figure
plots.metric_correlation_heatmap(per_patient_df)  # redundancy diagnostic

Empirical validation of the sample↔alarm bridge

The analytic bridge is validated by Monte Carlo. For each setting we synthesise a long per-window stream with a known per-window sensitivity s and specificity 1 − α plus seizures, run the AlarmPolicy, and measure the empirical alarm-sensitivity and FP/hr. We check (i) the empirical values land inside the analytic sample_to_alarm [lower, upper] bands, and (ii) the reverse alarm_to_sample recovers the true per-window s and specificity. Each seizure's SOP holds K = ceil(SOP / cadence) windows by construction, so the per-seizure detection bound is 1 − (1 − s)^K independent of prevalence — the soundness fix that replaced an earlier prevalence-shrunk K_eff which collapsed the upper bound to s at realistic low prevalence (empirical ≈ 1.0 vs that bound 0.5 → violated).

All four settings pass in both directions. Reproduce with python examples/06_bridge_validation.py (writes the figure to docs/bridge_validation.{png,pdf} and the table to examples/06_bridge_validation_out/); CI guards it via tests/examples/test_06_bridge_validation.py.

References

• Andrade I, Teixeira C, Pinto M (2024). On the performance of seizure prediction machine learning methods across different databases: the sample and alarm-based perspectives. Frontiers in Neuroscience. doi:10.3389/fnins.2024.1417748.
• Cook MJ et al. (2013). Lancet Neurology. doi:10.1016/S1474-4422(13)70075-9.
• Dan J et al. (2024). SzCORE. Epilepsia. doi:10.1111/epi.18113.
• Direito B et al. (2011). EPILAB. J Neurosci Methods. doi:10.1016/j.jneumeth.2011.06.022.
• Karoly PJ et al. (2017). Brain. doi:10.1093/brain/awx173.
• Kuhlmann L et al. (2018). Brain. doi:10.1093/brain/awy210.
• Maturana MI et al. (2020). Nature Communications. doi:10.1038/s41467-020-15908-3.
• Mormann F et al. (2007). Seizure prediction: the long and winding road. Brain. doi:10.1093/brain/awl241.
• Schulze-Bonhage A et al. (2020). Performance Metrics for Online Seizure Prediction. PMC7340210.

Part of SciTeX

scitex-seizure-metrics is part of SciTeX. Install via the umbrella with pip install scitex[seizure-metrics] to use as scitex.seizure_metrics (the seizure-evaluation surface re-exported from this peer; equivalent to scitex-ml[seizure] / scitex_ml.metrics.seizure for users who only want this slice without the rest of scitex-ml).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.