drift-or-shift

A reproducible research pipeline for distinguishing label shift from concept drift, and for showing exactly when a training-free logit offset is sufficient — and when it is not.

The experiments reproduce and extend the analysis in SSRN 6052514: start from synthetic two-class Gaussian data, apply the Bayes-optimal offset correction without retraining, then observe which metrics are invariant (ROC AUC) and which are not (PR-AUC, effective sample size). When the class-conditionals themselves move, the offset provably cannot recover the new posterior, and retraining becomes necessary.

Installation

# From source, for development
git clone https://github.com/DiogoRibeiro7/drift-or-shift.git
cd drift-or-shift
make install          # editable install + dev extras + pre-commit hooks

Without make:

python -m pip install -e ".[dev]"
python -m pre_commit install

Runtime install only:

python -m pip install .

Requires Python 3.10+. This installs the drift_or_shift package, the drift_shift_pipeline compatibility alias, the caliblab calibration benchmarking helpers, and the eleven dos-expN console scripts.

Quickstart

# Fast end-to-end sanity check (seconds)
python scripts/smoke_exp1.py --results-dir results/smoke

# A full experiment
dos-exp1 --pi-train 0.2 --pi-tests 0.05 0.2 0.5 --results-dir results

# Aggregate every run into a reviewable dashboard
python scripts/aggregate_results.py --results-dir results \
    --output reports/dashboard.md --figure reports/best_risk.png

Each run writes tables/, figures/, and a _summary.json into results/<experiment>/<utc-timestamp>/.

Core concepts

Label shift vs. concept drift

Offset correction

Logits are log-odds, so shifting the prior from π_train to π_test is a constant additive term:

offset = log( π_test · (1 − π_train) / (π_train · (1 − π_test)) )

Apply the offset before comparing against the cost-derived threshold log(c10 / c01); with default costs (c10 = c01 = 1) that threshold is zero. Because a constant offset preserves the ranking of scores, ROC AUC is invariant under label shift — while PR-AUC, which depends on prevalence, is not.

Experiments

Sources live in src/drift_or_shift/experiments/. Every script accepts --help.

Reproducibility

• All generators take an explicit seed; summaries record seeds, sizes, costs, and prevalence grids.
• Run-directory names and summary timestamps are UTC and timezone-aware.
• ExperimentConfig in drift_or_shift.experiments._common (or the CLI flags) pins a run's configuration.
• reproduce/ holds a separate calibration-benchmark replication driven by reproduce/config/default.yaml.

Tooling utilities

• drift_or_shift.drift_variants — inject covariance/feature shifts, label noise, and density-ratio scoring without rebuilding the generators.
• scripts/aggregate_results.py — collect every results/*/*_summary.json into a markdown dashboard; --fail-on-alerts exits non-zero when drift is detected.
• scripts/drift_alerts.py — flag feature_max_* statistics above threshold into reports/drift_alerts.csv.
• scripts/watch_results.py — poll results/ and refresh alerts + dashboard as new summaries land (--once for a single pass).

See docs/CLI_USAGE.md for full flag documentation.

Development

make check      # ruff + black + mypy + pytest, exactly what CI runs
make test-cov   # tests with an HTML coverage report
make build      # sdist + wheel, validated with twine
make help       # all targets

CI runs the suite on Python 3.10–3.13 on Linux, plus Windows and macOS spot checks, and verifies that the built wheel installs and runs in a clean environment.

See CONTRIBUTING.md for branching and PR conventions, and CHANGELOG.md for release history.

Project documents

Citation

If this code supports academic work, please cite the underlying paper (SSRN 6052514) and reference this repository:

@software{ribeiro_drift_shift_pipeline,
  author  = {Ribeiro, Diogo},
  title   = {drift-or-shift: label shift, offset correction and drift diagnostics},
  url     = {https://github.com/DiogoRibeiro7/drift-or-shift},
  version = {0.1.0}
}

License

Released under the MIT License.