deferguard 0.3.0

Turn any confidence signal into a calibrated act-vs-escalate decision with a distribution-free statistical guarantee.

DeferGuard

Turn "should I trust this enough to act alone" into a number you can defend, not a threshold you picked by feel.

DeferGuard is a small, framework- and domain-agnostic Python library that takes any confidence signal your AI system already produces — a logprob, a self-consistency vote, an LLM-judge score, an ensemble agreement fraction — and turns "should this act autonomously or go to a human" into a decision backed by a distribution-free statistical guarantee, calibrated on your own labeled data.

• Calibrate once: feed it a batch of (confidence_signal, was_it_correct) pairs from historical data.
• Decide at runtime: for each new case, DeferGuard returns act or escalate, plus a plain-language guarantee string.
• Recalibrate cheaply: when the underlying model, judge, or expert pool changes, recalibrate on a fresh small batch — no retraining of a separate "rejector" model.

It is not an LLM router, not a guardrail/moderation heuristic library, and not a hosted platform — it's a lightweight statistics layer (no embeddings, no vector store, no GPU, no model API calls required) that sits on top of a well-established but under-productized statistical result: conformal deferral / conformal risk control.

Status

Phase 1 (core calibration engine), Phase 2 (pluggable signal adapters + ConformGuard interoperability), and Phase 3 (multi-expert routing + conditional coverage) are done — see CHANGELOG.md for what's implemented and Roadmap below for what's next.

Roadmap

DeferGuard is built in phases; each is meant to be fully working and tested before the next begins.

• Phase 1 — core calibration engine (done): Calibrator with .fit(), .decide(), .save()/.load(), .recalibrate(); local persistence; a CLI to calibrate, check coverage, and audit past decisions; the coverage-validation suite and negative controls described below; explicit assumption checks (monotonicity, minimum-n, an exchangeability-shift proxy).
• Phase 2 — pluggable signal adapters + ConformGuard interoperability (done): a signals module (LogprobSignal, JudgeScoreSignal, EnsembleVoteSignal using linear opinion pooling across multiple agents); a ConformGuard adapter converting a real WrapResult / MultiCheckWrapResult decision into a DeferGuard signal; drift monitoring that tracks the live "act" error rate once delayed ground truth is available and warns when recalibration may be due; deferguard[signals] / deferguard[conformguard] install extras. See docs/signals_and_integrations.md.
• Phase 3 — multi-expert routing + conditional coverage (done): routing.SegregativityRouter, implementing Bary, Macq & Petit's training-free segregativity criterion (arXiv:2509.12573) to route an escalated case to a specific expert without retraining; core.GroupCalibrator, giving an error-rate guarantee that holds within caller-declared subgroups (Vovk 2012's conditional/Mondrian conformal prediction, arXiv:1209.2673), not just marginally overall; cost-aware calibration (Calibrator(..., max_escalation_rate=...)) respecting both an error-rate target and an escalation-rate budget, with an explicit, non-silent conflict signal when the two can't both be satisfied. See docs/multi_expert_and_conditional_coverage.md.
• Phase 4 — monitoring integrations, federation, streaming recalibration: OpenTelemetry span export of every decision for existing observability tools like Phoenix or Langfuse (not a competing dashboard); optional streaming/online recalibration for high-volume, fast-drifting deployments; optional, opt-in shared/federated calibration sets across teams.

Quickstart

pip install deferguard

Developing DeferGuard itself, or installing from source instead of PyPI? Use an editable install: pip install -e . (see Development below).

from deferguard import Calibrator, CalibrationExample

calibrator = Calibrator(alpha=0.05)  # target: at most 5% of "act" decisions are wrong

# calibration_set: historical (signal, correctness) pairs from your own domain
calibration_set = [
    CalibrationExample(signal=0.91, correct=True),
    CalibrationExample(signal=0.44, correct=False),
    # ... hundreds to thousands more, depending on alpha -- see below
]
calibrator.fit(calibration_set)
calibrator.save("calibration.json")

# --- at runtime ---
calibrator = Calibrator.load("calibration.json")
decision = calibrator.decide(signal=0.83)

if decision.action == "escalate":
    route_to_human(item)
else:
    proceed_autonomously(item)

print(decision.guarantee)
# "With probability at least 95.0% over this calibration draw, at most 5.0% of
#  ACT decisions are wrong, assuming the calibration set and live traffic are
#  exchangeable. Calibrated with Conformal Risk Control on n=940 examples on
#  2026-07-19."

See examples/ for three fuller, runnable scripts (support-bot escalation, moderation pipeline, multi-agent ensemble) using synthetic data — no API keys required.

How much calibration data do you need?

More than a naive "n ≥ 1/alpha" rule of thumb would suggest — DeferGuard's guarantee is a training-conditional one (see docs/understanding_the_guarantee.md), which is stronger and more useful than a plain average-case guarantee, but costs more data for a given alpha. If you're not sure your data is enough, ask:

from deferguard.core.validation import recommend_alpha
recommend_alpha(n=500)   # -> the tightest alpha your 500 examples can honestly support

or from the CLI: deferguard recommend-alpha 500. Calibrator.fit() raises InsufficientCalibrationDataError (with this same suggestion baked into the message) rather than silently producing a threshold that isn't backed by enough data.

Splitting your data across subgroups with GroupCalibrator (Phase 3) multiplies this requirement by roughly the number of groups — see docs/multi_expert_and_conditional_coverage.md for the specific numbers.

Coverage validation

This table is the project's actual proof, not illustrative numbers — produced by tests/coverage_validation/synthetic/test_known_distribution.py, which repeatedly (300 independent draws per row) calibrates against a known ground-truth process and checks the resulting threshold's exact, analytically-computed true selective risk against the target. Reproduce it yourself: pytest tests/coverage_validation/synthetic -v -s.

Well-separated signal (correctness concentrates sharply at high signal — required, asserted by CI):

alpha	n (calibration)	achieved	mean true risk	P(true risk ≤ alpha)	target (1 − alpha)	result
0.20	208	300/300	0.0315	1.000	0.800	PASS
0.10	507	300/300	0.0256	1.000	0.900	PASS
0.05	1222	300/300	0.0211	1.000	0.950	PASS
0.01	8281	300/300	0.0031	1.000	0.990	PASS

Moderately-separated signal (harder, more realistic case — informational, shown so the bound isn't just trivially loose; alpha=0.01 needs more calibration data than is practical to run in CI here and is omitted, not hidden — see docs/architecture.md):

alpha	n (calibration)	achieved	mean true risk	P(true risk ≤ alpha)	target (1 − alpha)	result
0.20	480	200/200	0.1078	1.000	0.800	PASS
0.10	1170	200/200	0.0442	1.000	0.900	PASS
0.05	2820	193/200	0.0161	1.000	0.950	PASS

mean true risk and P(true risk ≤ alpha) are both computed against the exact analytic selective risk of the known data-generating process, not an estimate from a finite held-out sample. Note that mean true risk sits closer to (while still safely under) the target than a first, buggier version of this table showed: an earlier select_threshold implementation stopped at the first act-set size in its search grid that satisfied alpha, instead of continuing to find the largest satisfying act-set (the documented design intent). That bug was always safe — every grid point covered by the underlying union bound is individually valid, and P(true risk ≤ alpha) was 1.000 both before and after — but it was needlessly conservative: on this same benchmark, mean escalation rate at alpha=0.05 dropped from 92.3% to 43.0% once fixed, i.e. the earlier version was sending more than 2x as many cases to a human as the guarantee actually required. See docs/architecture.md for the full account.

Negative controls (tests/coverage_validation/synthetic/test_negative_controls.py) — deliberately broken assumptions, confirmed to fail visibly rather than silently:

assumption broken	what was checked	result
Exchangeability	Calibrate on an easy population, evaluate the resulting threshold against a shifted, harder population	Guarantee measurably fails — true risk 0.639 vs. target alpha=0.1 (6.4× over), exactly as documented, not hidden
Monotonicity	Construct a signal anti-correlated with correctness	MonotonicityWarning fires during .fit(), and the resulting threshold's true risk exceeds alpha, confirming the warning isn't a formality

Understanding the guarantee — read this before you rely on it

The one-sentence version: "at most alpha% of ACT decisions are wrong" holds only if your calibration set and live traffic are exchangeable — statistically, "live cases look like a random draw from the same population the calibration set came from." If your model changes, your user base shifts, or your product adds a new use case, this can silently stop being true until you recalibrate. See docs/understanding_the_guarantee.md for the full, plain-language explanation — this is not optional reading.

CLI

deferguard calibrate data.csv --alpha 0.05 --output calibration.json
deferguard check calibration.json holdout.csv
deferguard audit audit.jsonl
deferguard recommend-alpha 500

Signals and integrations (Phase 2)

Optional helpers for turning a specific kind of raw input into the float decide() already accepts — logprobs, a judge score, multiple agents' verbalized probabilities, or a ConformGuard decision:

from deferguard.signals import LogprobSignal, JudgeScoreSignal, EnsembleVoteSignal
from deferguard.integrations import to_signal  # requires deferguard[conformguard]

signal = LogprobSignal.from_token_logprobs([-0.05, -0.12, -0.01])
signal = JudgeScoreSignal.from_scaled_score(8, score_range=(1, 10))
signal = EnsembleVoteSignal.aggregate([0.92, 0.88, 0.95])  # linear opinion pool
signal = to_signal(conformguard_wrap_result)               # WrapResult / MultiCheckWrapResult

decision = calibrator.decide(signal=signal)

Also in Phase 2: deferguard.telemetry.check_drift compares the live "act" error rate against your calibrated alpha target once delayed ground truth is available, and warns (never auto-recalibrates) when they've drifted apart. Full usage and the exact math behind EnsembleVoteSignal (with the paper equation it implements) are in docs/signals_and_integrations.md.

Documentation

• docs/architecture.md — module layout and the CRC-adaptation design decision (including a real correctness bug this project caught and fixed during Phase 1 development, and why)
• docs/understanding_the_guarantee.md — what the guarantee promises, what it doesn't, and what "training-conditional" means
• docs/choosing_a_confidence_signal.md — what makes a signal usable, and common pitfalls
• docs/signals_and_integrations.md — Phase 2 API reference: signals, integrations.conformguard_adapter, telemetry.drift
• docs/multi_expert_and_conditional_coverage.md — Phase 3 API reference: routing.SegregativityRouter, core.GroupCalibrator, cost-aware calibration
• docs/real_world_validation.md — real self-consistency validation against 5 locally-running open-weight models via Ollama, on both an easy and a deliberately harder benchmark; includes models where the signal wasn't monotonic, a model that lost calibratability entirely on the harder set, and one whose monotonicity reversed between the two runs
• docs/phase3_real_world_validation.md — real-model validation of GroupCalibrator, cost-aware calibration, and SegregativityRouter against 4 of the same local models; includes a real per-subgroup coverage violation that GroupCalibrator avoids, a genuine cost-aware conflict case, and an honestly-reported case where SegregativityRouter's real-model exercise turned out less informative than hoped

References

The statistical core is Conformal Risk Control, adapted (see docs/architecture.md for the full derivation, including a real bug the adaptation caught and fixed) into a training-conditional, selective-risk guarantee via a Bonferroni-corrected, simultaneous Clopper-Pearson bound.

• Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2024). Conformal Risk Control. ICLR 2024. arXiv:2208.02814 — the foundational finite-sample correction this project's core is built on.
• Verma, R., Barrejón, D., & Nalisnick, E. (2023). Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. arXiv:2210.16955
• Fang, Y., & Nalisnick, E. (2025). Learning to Defer with an Uncertain Rejector via Conformal Prediction. TMLR / NeurIPS 2025 Workshop on Bayesian Decision-making and Uncertainty. Reference implementation
• Bary, T., Macq, B., & Petit, L. (2025). No Need for Learning to Defer? A Training-Free Deferral Framework to Multiple Experts through Conformal Prediction. arXiv:2509.12573 — the training-free-recalibration principle this project follows (see "Recalibrate cheaply" above).
• Wang, M. F., et al. (2026). From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation. arXiv:2604.07667 — applies the same act-vs-escalate idea to multi-agent voting/debate; Definition 1's linear opinion pool is what signals.EnsembleVoteSignal implements (see signals/ensemble.py for the cited equation and what's deliberately out of scope).

Development

pip install -e ".[dev]"
pytest tests/unit tests/property                # fast, default CI gate
pytest tests/coverage_validation/synthetic       # the coverage-guarantee proof
pytest tests/coverage_validation/live -m live    # opt-in, real API calls
pytest tests/integration                         # CLI end-to-end

See CONTRIBUTING.md.

License

Apache 2.0 — see LICENSE.