probcal

Probability calibration you can put in front of a validator: numpy-only methods, metrics, and diagnostics for binary classifiers — built for regulated PD models, fully general in practice.

What the wedge is, concretely:

• Regulated-PD first. Logit-scale diagnostics that keep 1% readable, per-grade regulatory backtests (binomial, Jeffreys), and a central-tendency adjustment that ships as an auditable LogitOffset stage — never a silent refit.
• Audit trail everywhere. Every method explains its own parameters (interpret()); every fitted object serializes to versioned JSON (never pickle) with provenance fingerprints; every refusal names the reason and the alternative — no silent clamps, no silent approximations.
• Exact inverse maps. "PD ≤ 2%" translates to a raw-score threshold, scorecard point cut-offs, or a counterfactual target, exactly (interval_inverse / point_inverse / Chain) — the contract recourse engines like treecf build on.
• Anytime-valid monitoring. probcal.monitor watches deployed calibration with e-processes: the alarm keeps its type-I guarantee at every look, and reports whether a re-offset is enough or a re-fit is due.
• numpy-only core. import probcal pulls numpy and nothing else — enforced by a test. scikit-learn/optbinning/treecf adapters are opt-in extras.

Start here: the executed end-to-end notebook takes one rare-event portfolio from GBM baseline to reliability diagnosis, selection with CIs, per-grade backtests, offsetting, threshold translation, a counterfactual, monitoring, and JSON round-trips.

probcal covers the binary calibration literature (Platt, temperature, beta, isotonic, centered isotonic, histogram binning, scaling-binning, BBQ, ENIR, Venn–Abers, spline), an extensive metric catalog with bootstrap CIs, automatic method selection under nested validation, and prefit/cv data flows.

Status: released on PyPI, beta. The API is stable enough to build on; breaking changes bump the minor version until 1.0 (see API stability in the docs). Serialized artifacts have a stronger promise: every 0.x release reads schema 1, enforced by golden files in CI.

Installation

pip install probcal            # runtime: numpy only
pip install "probcal[viz]"     # + matplotlib for probcal.plots
pip install "probcal[sklearn]" # + scikit-learn for probcal.sklearn adapters

Development setup (tests, lint, type-check):

git clone https://github.com/wlazlod/probcal && cd probcal
uv sync --extra dev

Quickstart

from probcal import BetaCalibrator, make_pd_portfolio
from probcal.metrics import calibration_guardrails

port = make_pd_portfolio(n=8000, random_state=42)   # synthetic 3% PD portfolio

g_before = calibration_guardrails(port.y, port.scores)
print(f"before: slope={g_before.slope:.3f}  intercept={g_before.intercept:+.3f}  ok={g_before.all_ok}")

cal = BetaCalibrator().fit(port.scores, port.y)
p = cal.predict_proba(port.scores)

g_after = calibration_guardrails(port.y, p)
print(f"after:  slope={g_after.slope:.3f}  intercept={g_after.intercept:+.3f}  ok={g_after.all_ok}")
print()
print(cal.interpret())

Output:

before: slope=0.968  intercept=-0.765  ok=False
after:  slope=1.000  intercept=+0.000  ok=True

Interpretation[BetaCalibrator]
parameter  value
---------  --------
a          0.875054
b          1.58922
c          -1.15227
- a = 0.875: sensitivity near s -> 0; a < 1 raises the smallest probabilities (model was overconfident in the low tail), a > 1 deepens them
- b = 1.589: sensitivity near s -> 1; the mirrored reading for the high tail
- c = -1.152: base-rate shift of -1.152 log-odds, odds factor 0.316
- identity map corresponds to (a, b, c) = (1, 1, 0)
- a != b (gap -0.714): asymmetric tail distortion that no symmetric (Platt/temperature) map could express

Automatic selection, model wrapping, offsetting, and threshold translation:

from probcal import CalibratedModel, CalibratorSelector, PlattCalibrator

sel = CalibratorSelector().fit(s_cal, y_cal)             # nested CV, log-loss criterion
wrapped = CalibratedModel(model, PlattCalibrator(), flow="prefit").fit(X_cal, y_cal)
wrapped.offset_to(target_mean=0.031)                     # auditable central-tendency stage
lo_z, hi_z = wrapped.interval_inverse(0.0, 0.02, space="logit")   # "PD <= 2%" in raw margins

Why probcal

¹ betacal, venn-abers, ml-insights. ² prdm0/probcal (P. R. Diniz Marinho), unaffiliated — see the FAQ. Verified against v0.2.0, 2026-08-08. ³ Platt, temperature, beta, isotonic, histogram binning; its multiclass methods (Dirichlet, vector scaling, one-vs-rest) are out of probcal's binary scope.

Serialization

Every fitted object round-trips through versioned, human-readable JSON — never pickle (auditable; loading executes no code):

cal.to_json("beta.json")
loaded = BetaCalibrator.from_json("beta.json")   # bit-identical predictions
cal.fingerprint()                                # sha-256 provenance id

Compatibility promise: every 0.x release reads schema 1, enforced by committed golden files in CI; schema bumps ship only with a converter. Details: the Serialization concepts chapter.

Calibrators at a glance

Performance note: the ICI family (ici/e50/e90/emax) shares one LOESS fit anchored to grid_size=512 quantile points instead of refitting at every observation — the same device R's stats::lowess uses via its delta parameter (fit at spaced points, interpolate the rest) — and smooth_ece pre-aggregates its residual measure onto bins=8192 cells before the bandwidth bisection — for every n as of this release (0.1.3 ran the exact path for n ≤ 8192: ~1s at n=4000 on this host; now 1–6ms for all n up to 10⁵ and ~43ms at n=10⁶, where the O(n) pre-binning dominates). Measured on this host: ici at n=50,000 dropped from 192.2s (v0.1.2) to 1.2s, and loess(grid_size=512) now fits n=1,000,000 points in under 30s. grid_size=None and bins=None recover the exact pre-0.1.3 values and cost, so nothing is lost for portfolios small enough to afford it. Still numpy-only; Rust acceleration remains out of scope unless a future workload demands it.

Documentation

Built with mkdocs-material; run locally with uv run mkdocs serve. Start with Getting started, then the Concepts chapters — the package's theoretical foundation — and the executed PD calibration walkthrough notebook. The Visualization chapter is a gallery of every plot, regenerated deterministically by docs/scripts/generate_figures.py.

License

MIT. See LICENSE. GPL-licensed R packages are used as conceptual references only; no GPL code is included.