probcal

Probability calibration for binary classifiers, built for regulated PD models. The runtime is numpy only. The calibration maps are the standard ones; what probcal adds is the evidence a validator asks for next, which scikit-learn does not produce:

• Logit-scale diagnostics that keep 1% readable. calibration_guardrails reports slope, intercept, and Spiegelhalter's test in log-odds, and every reliability curve draws on the logit axis, so a 3% portfolio is not crushed into the bottom-left corner.
• An auditable offset stage. A central-tendency adjustment is a LogitOffset object, not a refit: audit_report() records the mean before and after, the guardrails before and after, a timestamp, and a fingerprint.
• Per-grade regulatory backtests. binomial_grade_test and jeffreys_grade_test give the traffic-light table per rating grade, and jeffreys_upper_bands turns the same posteriors into a masterscale.

Installation

pip install probcal            # runtime: numpy only
pip install "probcal[viz]"     # + matplotlib for probcal.plots
pip install "probcal[sklearn]" # + scikit-learn for probcal.sklearn adapters

Beta on PyPI. Breaking changes bump the minor version until 1.0 (see API stability in the docs); serialized artifacts have a stronger promise, every 0.x release reads schema 1. Development setup:

git clone https://github.com/wlazlod/probcal && cd probcal
uv sync --extra dev

Quickstart

The three things above, in one run. The calibrator is fitted on one synthetic 3% portfolio and every number below is measured on a second, held-out draw; fitting and scoring the same rows would make the "after" line an identity (see Data splitting in the docs):

import numpy as np
from probcal import BetaCalibrator, LogitOffset, make_pd_portfolio
from probcal.metrics import calibration_guardrails, jeffreys_grade_test

cal_set = make_pd_portfolio(n=8000, random_state=42)   # synthetic 3% PD portfolio
test = make_pd_portfolio(n=8000, random_state=1)       # held-out draw, same distortion

# 1. Logit-scale diagnostics: slope, intercept, Spiegelhalter, before and after
g0 = calibration_guardrails(test.y, test.scores)
print(f"before: slope={g0.slope:.3f}  intercept={g0.intercept:+.3f}  ok={g0.all_ok}")
cal = BetaCalibrator().fit(cal_set.scores, cal_set.y)
p = cal.predict_proba(test.scores)                    # evaluated on the held-out set
g1 = calibration_guardrails(test.y, p)
print(f"after:  slope={g1.slope:.3f}  intercept={g1.intercept:+.3f}  ok={g1.all_ok}")

# 2. An auditable offset: re-anchor to a 3.5% policy PD and record what it cost
off = LogitOffset(target_mean=0.035).fit(p)
print(off.audit_report(test.y, p))
p_final = off.transform(p)

# 3. Per-grade regulatory backtest on a fixed PD masterscale
edges, labels = np.array([0, 0.01, 0.02, 0.05, 0.10, 1.0]), np.array(list("ABCDE"))
grades = labels[np.searchsorted(edges, p_final, side="right") - 1]
res = jeffreys_grade_test(test.y, p_final, grades)
for g, n, k, pd_, light in zip(res.grades, res.n, res.k, res.pd, res.light):
    print(f"grade {g}: n={n:5d}  defaults={k:3d}  PD={pd_:.4f}  {light}")

Output:

before: slope=0.901  intercept=-0.743  ok=False
after:  slope=0.922  intercept=+0.021  ok=True
AuditReport(delta=+0.1238, odds factor 1.1317, fitted <timestamp>)
  portfolio mean: 0.03126 -> 0.03500
  slope:          +0.922 -> +0.922
  intercept:      +0.021 -> -0.103
  spiegelhalter p 0.460 -> 0.271
  guardrails ok:  True -> False
grade A: n= 1696  defaults=  7  PD=0.0063  green
grade B: n= 2043  defaults= 22  PD=0.0146  green
grade C: n= 2684  defaults= 97  PD=0.0319  green
grade D: n= 1114  defaults= 70  PD=0.0681  green
grade E: n=  463  defaults= 59  PD=0.1685  green

The audit report says what the policy offset cost: the intercept guardrail now fails by exactly the applied shift, on the record. The executed end-to-end notebook takes a rare-event portfolio through selection, backtests, offsetting, threshold translation, and monitoring.

Measured against scikit-learn and netcal

Taiwan credit-card default (OpenML default-of-credit-card-clients, n=30,000, 22.1% default rate). A scorecard base model with integer points gives the scores a deployed scorecard has: 161 distinct values across 7,500 calibration obligors, largest tie block 164. Grades come from an 8-band PD masterscale applied to each method's own calibrated PD; grade A (PD under 3%) holds 12 to 46 obligors depending on the calibrator, and netcal BBQ never reaches it. Evaluation on 7,500 held-out obligors with bootstrap CIs; the grade column is the number of grades passing the Jeffreys backtest at the 5% level.

The calibration maps agree: probcal's Platt equals sklearn's sigmoid and probcal's isotonic equals sklearn's isotonic to four decimals, and the beta row matches netcal and betacal. On a plain scorecard the map is not where the libraries differ; the diagnostics, the offset audit, and the per-grade table above are. NaN scores are rejected with a named error, never imputed. Full method list, five more datasets from 1.5% to 30% event rate, and the reproducible script: Benchmarks.

Three calibrators, and the rest

PlattCalibrator, BetaCalibrator, and IsotonicCalibrator are the production path: the quickstart and the benchmark use them, each has exact inverse maps, interpret(), and JSON serialization pinned by golden files. Ten more (temperature, centered isotonic, histogram binning, scaling-binning, BBQ, ENIR, Venn–Abers IVAP and CVAP, spline, segmented) and the nested-CV CalibratorSelector are in the calibrator catalog, with monotonicity, inverses, data appetite, and fit cost per row. The 47-symbol metric catalog, with which metrics are safe to select on, is in Metrics and tests.

Serialization

Every fitted object round-trips through versioned, human-readable JSON, never pickle (auditable; loading executes no code):

cal.to_json("beta.json")
loaded = BetaCalibrator.from_json("beta.json")   # bit-identical predictions
cal.fingerprint()                                # sha-256 provenance id

Compatibility promise: every 0.x release reads schema 1, enforced by committed golden files in CI; schema bumps ship only with a converter. Details: the Serialization concepts chapter.

Documentation

Built with mkdocs-material; run locally with uv run mkdocs serve. Start with Getting started, then the Concepts chapters — the package's theoretical foundation. The Visualization chapter is a gallery of every plot, regenerated deterministically by docs/scripts/generate_figures.py; the CORP reliability diagram, MCB-DSC plane, and score decomposition have their own CORP and score decomposition chapter.

License

MIT. See LICENSE. GPL-licensed R packages are used as conceptual references only; no GPL code is included.