FPRCal

Stable low-FPR calibration for detection model scores across model releases, with a fixed log-scale interpretability contract (0.5 = 0.1% FPR, 0.7 = 0.01%, 0.85 = 0.001%).

Why

Raw scores from ML detectors drift across model releases and are not comparable across detector categories. Downstream product rules break on every retrain. This package calibrates detector scores to FPR on benign traffic, then applies a fixed log-scale transform so every calibrated value has the same FPR meaning across model versions and detector categories.

Install

python -m pip install fprcal

Requires Python 3.12+, numpy, scikit-learn, joblib.

For an editable development install, use python -m pip install -e ".[dev]".

Usage

from fprcal import fit_calibration_pipeline
import joblib

pipeline = fit_calibration_pipeline(benign_scores, n_knots=10000)
joblib.dump(pipeline, "calibration.pkl")

# In production:
pipeline = joblib.load("calibration.pkl")
calibrated = pipeline.predict(raw_scores.reshape(-1, 1))

The first-pass FPR-to-threshold spline uses Filliben median-centered plotting positions by default. Pass plotting_position="mean" to use the mean-centered k/(n+1) positions instead.

Demo

Reproduce the evaluation figure on the Credit Card Fraud Detection dataset (OpenML, 284K rows, 0.172% positives):

# One-time: download the data, train both detectors, save holdout scores
python examples/generate_credit_card_roc.py

# Fit on the calibration subset and evaluate its held-out complement
python examples/calibration_demo.py

Writes examples/credit_card_validation.png. See examples/credit_card_readme.md for dataset provenance.

License

Apache License 2.0. See LICENSE.

Maintainers can find the release procedure in RELEASING.md.