CalibRoute
CalibRoute is a lightweight Python toolkit for confidence evaluation and
uncertainty-aware routing. It converts model predictions into three actions:
accept, human_review, or abstain.
Try it in your browser: the interactive demo on Hugging Face Spaces runs the 0.5.0 package with Pyodide on the Financial NER and ShiftGuard examples, or on a CSV you upload. Nothing leaves your browser.
Features
- Confidence calibration and error-ranking metrics
- Risk-coverage analysis
- Validation-only threshold fitting
- Tie-aware threshold fitting and independent holdout risk assessment
- Batch-size-aware confidence-shift detection
- Financial NER sentence- and entity-level output adapters
- ShiftGuard agent-replay adapter and offline confidence audit
- CSV and JSONL support
- Dependency-free Python API and CLI
Install
Requires Python 3.10 or newer.
Install with python -m pip install calibroute-ai.
To obtain example scripts, clone the repository:
git clone https://github.com/Garyouki/calibroute.git
cd calibroute
Use the installed release for the walkthroughs. For source development, install
the checkout with python -m pip install -e ..
The distribution name is calibroute-ai; the Python import and command are
calibroute. No API key or model service is needed.
Version compatibility
Release 0.5.0 includes the 0.4.0 features below and hardens holdout validation. When upgrading, give each holdout prediction a stable, unique ID. Validation uses the policy's saved confidence level unless explicitly overridden.
Release 0.4.0 adds entity-level NER, ShiftGuard, and comparison reports to
the PyPI package. Upgrade with python -m pip install --upgrade calibroute-ai.
| Capability | PyPI 0.3.0 | PyPI 0.4.0 |
|---|---|---|
audit, fit, validate, route |
Available | Available |
Three-rule comparison report: compare |
Not included | Available |
| Tie-aware thresholds, holdout assessment, and confidence-shift routing | Available | Available |
Financial NER sentence adapter: convert-financial-ner |
Available | Available |
Financial NER entity adapter: convert-financial-ner-entities |
Not included | Available |
ShiftGuard replay adapter: convert-shiftguard |
Not included | Available |
| Document-review walkthrough | Compatible; obtain script from GitHub | Compatible; obtain script from GitHub |
| Financial NER sentence case study | Compatible; obtain script/data from GitHub | Compatible; obtain script/data from GitHub |
| ShiftGuard case study | Not compatible | Compatible; obtain script/data from GitHub |
For reproducible installation, use python -m pip install calibroute-ai==0.5.1.
Example scripts and data are available in the repository; installing the wheel
does not install the example folders. Check out tag v0.5.1 for examples matching
this release; GitHub main may receive further development changes.
To check the active environment, run python -m calibroute.cli --help and
python -m pip show calibroute-ai. Check the listed commands and installation location; record
git rev-parse HEAD when reporting results from a source checkout.
Quick start
To compare all-accept, a fixed threshold, and CalibRoute on the same labeled batch, see the one-command comparison guide (0.4.0 or newer).
New here? Follow the document-review walkthrough: one offline script demonstrates development data, a separate holdout, and routing using the published 0.5.0 package. For your own exports, follow the CSV onboarding walkthrough for explicit column mapping, clean installation, and frozen-policy comparison. The column helper is a repository example. Use the matching tag and record the checkout commit when reporting results.
For measured limitations of confidence-shift monitoring, see the synthetic shift study, including false alarms, label-only accuracy losses, and additional review workload.
# Audit labeled predictions
calibroute audit \
--input examples/validation.csv \
--output examples/audit.md
# Fit an acceptance policy on validation data
calibroute fit \
--input examples/validation.csv \
--max-risk 0.20 \
--min-coverage 0.25 \
--risk-method empirical \
--output examples/policy.json
# Route a new prediction batch
calibroute route \
--input examples/production_batch.csv \
--policy examples/policy.json \
--output examples/decisions.csv \
--summary examples/routing-summary.json
Input format
| Field | Required | Description |
|---|---|---|
id |
required for validate |
Unique prediction identifier. record_id is also accepted. |
confidence |
yes | Number between 0 and 1 |
correct |
audit, fit, validate, and compare |
Boolean or 1/0 label |
domain |
no | Evaluation slice or deployment domain |
Additional columns are preserved as metadata.
The default fit searches thresholds using pointwise 95% Clopper-Pearson bounds. Threshold selection on the same labels does not provide a 95% guarantee for the selected policy. Freeze the policy, then assess it on a separate IID holdout:
calibroute validate --input holdout.csv --policy examples/policy.json --output holdout-report.json
Provide an explicit, unique ID for each holdout prediction. Assign the ID to
the underlying prediction before combining exports so duplicates can be
detected. The bound assumes independent IID holdout units.
validate uses the confidence level saved in the policy unless you pass
--confidence-level. Empirical fitting reports observed risk and stores
null in risk_upper_bound.
Exit code 0 means the holdout upper bound meets the declared risk limit. Exit code 1 means it does not, including when there are zero accepted samples. Exit code 2 indicates invalid input. Use an untouched holdout and interpret the result for its sampled distribution. See statistical scope.
Python API
from calibroute import PredictionRecord, fit_policy, route_batch
validation = [
PredictionRecord("a", 0.98, True),
PredictionRecord("b", 0.82, True),
PredictionRecord("c", 0.55, False),
]
policy = fit_policy(validation, max_risk=0.10, min_coverage=0.50,
risk_method="empirical") # Tiny illustrative sample only.
decisions, summary = route_batch(
[PredictionRecord("new", 0.74)],
policy,
)
Limitations
CalibRoute is an evaluation and routing tool, not a safety certification. Thresholds should be revalidated after changes to the model, task, prompt, or deployment distribution. See design principles for details.
Development
python -m unittest discover -s tests -v
See CONTRIBUTING.md and ROADMAP.md.
Tried it on a task? Share a usage report,
including unsuccessful trials or integration difficulties.
Release preparation is documented in releasing.
The Financial NER case study demonstrates
the adapter and cross-domain failure pattern on 2,098 derived prediction rows.
The ShiftGuard agent example audits 384 frozen
proposals, keeps task templates separate, and reports when no confidence
threshold meets the declared empirical risk target. Use CalibRoute 0.4.0 or newer
for the convert-shiftguard command.
License
MIT. See LICENSE.
Metadata
Release files for calibroute-ai 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| calibroute_ai-0.5.1.tar.gz | 362.1 kB | With provenance | Details for calibroute_ai-0.5.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| calibroute_ai-0.5.1-py3-none-any.whl | Python 3 | none | any | Details for calibroute_ai-0.5.1-py3-none-any.whl |
Total release size: 388.5 kB
Release files / calibroute_ai-0.5.1.tar.gz
| Download URL | calibroute_ai-0.5.1.tar.gz |
|---|---|
| Size | 362.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
58d2c9075894078bb1f3b5be5af9960be8ee270a45bd3067f17570150010a19b
|
|
BLAKE2b-256 checksum How to use checksums |
523db030e1e15527af9d77663e9883d3c9f8c160cbe09a2859de8810bca4e5d4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / calibroute_ai-0.5.1-py3-none-any.whl
| Download URL | calibroute_ai-0.5.1-py3-none-any.whl |
|---|---|
| Size | 26.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e1198850f44f84fd7585df4cc6650573bb939a52e912a787f44a1e7d8ce65988
|
|
BLAKE2b-256 checksum How to use checksums |
1faf17cfe3a039d98881a4fc8fec3dd7b6f9f5fb3447d388411d989ab901b32d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for calibroute-ai 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for calibroute_ai-0.5.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.