Fiducio

Fiducio is a model-agnostic Python library of post-hoc calibrators for 2D and 3D semantic segmentation. Give it the logits or probabilities of any segmentation model and a labelled calibration set to fit a probability transform without retraining the segmentation model. It implements temperature, vector, matrix and Dirichlet scaling, the translation-invariant matrix-scaling variant (MSc), and the class-conditional (CDC, CMSap, CMSop) calibrators introduced in our paper Rethinking Post-Hoc Calibration in Semantic Segmentation, accepted at TMLR. Whether the segmentation is preserved depends on the calibrator; see the guarantees below.

It works with PyTorch U-Net, nnU-Net, SegFormer or any model that produces logits: Fiducio only ever sees (B, C, *spatial) tensors and never needs to know the architecture.

Top-1 reliability diagrams on held-out synthetic data (regenerate with examples/readme_figure.py).

Install

Fiducio is available on PyPI:

pip install fiducio

or, in a uv application:

uv add fiducio

Requires Python ≥ 3.10 and PyTorch ≥ 2.10 (a security floor, see installation). The core install depends only on numpy and torch. Optional extras: fiducio[plots] (matplotlib reliability diagrams), fiducio[docs], fiducio[dev], fiducio[all].

To install the development version from GitHub:

pip install "git+https://github.com/fiducio-ai/Fiducio.git"

For development from a checkout, use the committed uv lockfile:

uv sync --locked
uv run --no-sync pytest

This uses CPU PyTorch for development. See CONTRIBUTING.md for the pinned uv version, documentation and build commands.

Quick start

import torch
from fiducio import TemperatureScaling, load_calibrator

# From a held-out, labelled calibration set:
torch.manual_seed(0)
logits = torch.randn(8, 4, 64, 64)  # (B, C, H, W) model logits
labels = torch.randint(0, 4, (8, 64, 64))  # (B, H, W) labels

calibrator = TemperatureScaling(input_type="logits", device="cpu").fit(logits, labels)

# Apply to new predictions (returns calibrated probabilities):
probs = calibrator.transform(torch.randn(2, 4, 64, 64))

# Save and reload (loads on CPU by default):
calibrator.save("calibrator.pt")
calibrator = load_calibrator("calibrator.pt")

The random tensors demonstrate the API. Measure calibration on separate test cases; see the Quickstart.

Every calibrator exposes the same API: fit(predictions, targets, mask=None), transform, predict_proba, fit_transform, save, and the top-level load_calibrator.

Accepted tensor shapes

The class axis is always dimension 1.

Inputs may be logits or probs (set input_type). Binary segmentation is two channels (C = 2). Floating class-distribution targets, including MHR, have the same shape as predictions. Use mask to exclude soft targets; ignore_index applies to hard labels.

Calibrators

This is a summary; the canonical table, with parameters and API links, is in Calibrators.

¹ In exact arithmetic; in float32 the mixture can resolve near-ties differently once it collapses towards the uniform component.

Translation-invariant means the output is unchanged when the same constant is added to every input logit of a voxel.

Every calibrator exposes decision_function (calibrated logits, or log-probabilities for isotonic calibration). The nine parametric methods have a configurable optimizer ("adam" default, or "lbfgs"); isotonic fitting uses weighted PAVA instead.

Early stopping

Adam-fitted calibrators support the paper's recipe of early stopping on a validation set (with an optional learning-rate decay on plateaus):

from fiducio import OrderPreservingMatrixScaling

calibrator = OrderPreservingMatrixScaling(max_iter=2000, patience=20, lr_patience=10)
calibrator.fit(
    logits,
    labels,
    val_predictions=val_logits,
    val_targets=val_labels,  # held out from both
)

The iterate with the best validation NLL is kept. The default fixed budget (max_iter) is short, so it can underfit expressive calibrators; prefer a larger max_iter together with patience for CDC, CMSap and CMSop. This reproduces only the Adam + validation-NLL early-stopping recipe; the paper's case batching, class weighting, schedules, clipping and hyperparameter search are not part of the library (see the implementation guide).

Paper method mapping

LTS is not included in this release; this table is not the complete experimental method inventory.

CDC, CMSap and CMSop are class-conditional: they fit one affine map per uncalibrated top class, optimized jointly in one cross-entropy loss. Regularization applies to each expert's induced affine map in the common logit space. Set independent_experts=True to fit each expert separately. See Calibrators for the regularization formula and method-specific limits.

Calibration metrics are included: negative_log_likelihood, expected_calibration_error (ECE, population-weighted), unweighted_calibration_error (ACE, unweighted over the same bins — doesn't let a sparsely populated bin get drowned out by a large one), brier_score, reliability_curve, plus an optional reliability-diagram plot (fiducio.plots.reliability_diagram, needs fiducio[plots]).

Before calibrating, fiducio.diagnose(logits, labels) returns a ReliabilityReport describing the model on the calibration set: class balance, global calibration with bootstrap intervals over images, per-class calibration, and images or classes where the Dice is close to zero. It does not print by itself; call print(fiducio.diagnose(...)) to read it (guide). Its intervals are not the paper's hierarchical-bootstrap tests.

BA-ECE (boundary-aware ECE), per-case metric aggregation, the DSC/flip-rate audit and the paper's paired hierarchical-bootstrap tests are not included; the shipped metrics are top-1 pooled point estimates. unweighted_calibration_error is the unweighted mean over non-empty uniform bins (not the adaptive ACE of other work) and is also available under the backwards-compatible alias average_calibration_error.

Decision and ordering guarantees hold in exact arithmetic and are subject to floating-point ties and rounding. They do not cover thresholded or post-processed segmentations.

Multi-rater calibration

TwinTrack (Kirscher et al., MIDL 2026, short paper) motivates binary isotonic calibration towards the mean human response (MHR). Fiducio also extends its nine parametric methods to accept class distributions. Missing annotations are excluded from the vote counts.

from fiducio import BinaryIsotonicCalibration, mean_human_response

mhr, annotated = mean_human_response(annotations, num_classes=2)  # (B,C,*spatial)
cal = BinaryIsotonicCalibration(input_type="probs").fit(probabilities, mhr, mask=annotated)
calibrated = cal.transform(test_probabilities)

The isotonic map is binary, with at most 250 bins and weighted PAVA. It may change a segmentation at threshold 0.5. Its tie handling and interpolation are Fiducio conventions; exact TwinTrack reproduction is not claimed. All fits remain in memory. See the multi-rater guide and examples/multi_rater_calibration.py for validation and the distinction between ECE against MHR and mean ECE against individual raters.

Documentation

The paper project page presents the methods, interactive illustrations and published results.

The paper implementation guide records method mapping, numerical-reference coverage, fitting limits and the remaining steps needed to reproduce the paper's experiments. For ensemble inputs, see examples/ensemble_pooling.py.

Results in the paper

The paper compares these calibrators on real segmentation benchmarks. The reliability diagrams below (paper appendix) show the uncalibrated ensemble pooling (p0, p̄) against the best calibrator per dataset; the exact protocol, datasets and metrics are in the paper. This repository ships the calibrators, not the fitted models or benchmark pipelines, so it does not regenerate these figures (see the implementation guide).

Full guide and API reference: https://fiducio-ai.github.io/Fiducio/

• Installation
• Quickstart
• Concepts
• Multi-rater calibration
• Calibrators
• Reliability diagnostics
• Benchmark
• Examples
• Tutorial (Oxford Pets)
• Plotting

Citation

Fiducio accompanies the paper Rethinking Post-Hoc Calibration in Semantic Segmentation (Kirscher et al., Transactions on Machine Learning Research, 2026; OpenReview, preprint: arXiv:2607.01902). If you use Fiducio in your research, please cite it using the metadata in CITATION.cff, or:

@article{kirscher2026rethinking,
  title   = {Rethinking Post-Hoc Calibration in Semantic Segmentation},
  author  = {Kirscher, Tristan and Kahl, Kim-Celine and Kovacs, Balint and
             Rokuss, Maximilian and Maier-Hein, Klaus and Coubez, Xavier and
             Meyer, Philippe and Faisan, Sylvain},
  journal = {Transactions on Machine Learning Research},
  issn    = {2835-8856},
  year    = {2026},
  url     = {https://openreview.net/forum?id=xwNoSNxgxV}
}

For work using calibration to MHR or the isotonic approach, also cite TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation (Kirscher et al., MIDL 2026, short paper):

@misc{kirscher2026twintrack,
  title         = {TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation},
  author        = {Kirscher, Tristan and Ertl, Alexandra and Maier-Hein, Klaus and
                   Coubez, Xavier and Meyer, Philippe and Faisan, Sylvain},
  year          = {2026},
  eprint        = {2604.15950},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {Accepted at MIDL 2026, short paper},
  url           = {https://arxiv.org/abs/2604.15950v2}
}

Status

Fiducio is beta (0.x): usable and tested, with an API that may still change before 1.0. It is not certified for safety-critical or clinical use.

License

Apache-2.0.