Skip to main content

CalibRoute

CalibRoute is a lightweight Python toolkit for confidence evaluation and uncertainty-aware routing. It converts model predictions into three actions: accept, human_review, or abstain.

Features

  • Confidence calibration and error-ranking metrics
  • Risk-coverage analysis
  • Validation-only threshold fitting
  • Tie-aware threshold fitting and independent holdout risk assessment
  • Batch-size-aware confidence-shift detection
  • Financial NER sentence- and entity-level output adapters
  • ShiftGuard agent-replay adapter and offline confidence audit
  • CSV and JSONL support
  • Dependency-free Python API and CLI

Install

Requires Python 3.10 or newer.

Install with python -m pip install calibroute-ai.

To obtain example scripts, clone the repository:

git clone https://github.com/Garyouki/calibroute.git
cd calibroute

Use the installed release for the walkthroughs. For source development, install the checkout with python -m pip install -e ..

The distribution name is calibroute-ai; the Python import and command are calibroute. No API key or model service is needed.

Version compatibility

Release 0.5.0 includes the 0.4.0 features below and hardens holdout validation. When upgrading, give each holdout prediction a stable, unique ID. Validation uses the policy's saved confidence level unless explicitly overridden.

Release 0.4.0 adds entity-level NER, ShiftGuard, and comparison reports to the PyPI package. Upgrade with python -m pip install --upgrade calibroute-ai.

Capability PyPI 0.3.0 PyPI 0.4.0
audit, fit, validate, route Available Available
Three-rule comparison report: compare Not included Available
Tie-aware thresholds, holdout assessment, and confidence-shift routing Available Available
Financial NER sentence adapter: convert-financial-ner Available Available
Financial NER entity adapter: convert-financial-ner-entities Not included Available
ShiftGuard replay adapter: convert-shiftguard Not included Available
Document-review walkthrough Compatible; obtain script from GitHub Compatible; obtain script from GitHub
Financial NER sentence case study Compatible; obtain script/data from GitHub Compatible; obtain script/data from GitHub
ShiftGuard case study Not compatible Compatible; obtain script/data from GitHub

For reproducible installation, use python -m pip install calibroute-ai==0.5.0. Example scripts and data are available in the repository; installing the wheel does not install the example folders. Check out tag v0.5.0 for examples matching this release; GitHub main may receive further development changes.

To check the active environment, run python -m calibroute.cli --help and python -m pip show calibroute-ai. Check the listed commands and installation location; record git rev-parse HEAD when reporting results from a source checkout.

Quick start

To compare all-accept, a fixed threshold, and CalibRoute on the same labeled batch, see the one-command comparison guide (0.4.0 or newer).

New here? Follow the document-review walkthrough: one offline script demonstrates development data, a separate holdout, and routing using the published 0.5.0 package. For your own exports, follow the CSV onboarding walkthrough for explicit column mapping, clean installation, and frozen-policy comparison. The column helper is a repository example. Use the matching tag and record the checkout commit when reporting results.

For measured limitations of confidence-shift monitoring, see the synthetic shift study, including false alarms, label-only accuracy losses, and additional review workload.

# Audit labeled predictions
calibroute audit \
  --input examples/validation.csv \
  --output examples/audit.md

# Fit an acceptance policy on validation data
calibroute fit \
  --input examples/validation.csv \
  --max-risk 0.20 \
  --min-coverage 0.25 \
  --risk-method empirical \
  --output examples/policy.json

# Route a new prediction batch
calibroute route \
  --input examples/production_batch.csv \
  --policy examples/policy.json \
  --output examples/decisions.csv \
  --summary examples/routing-summary.json

Input format

Field Required Description
id required for validate Unique prediction identifier. record_id is also accepted.
confidence yes Number between 0 and 1
correct audit, fit, validate, and compare Boolean or 1/0 label
domain no Evaluation slice or deployment domain

Additional columns are preserved as metadata.

The default fit searches thresholds using pointwise 95% Clopper-Pearson bounds. Threshold selection on the same labels does not provide a 95% guarantee for the selected policy. Freeze the policy, then assess it on a separate IID holdout:

calibroute validate --input holdout.csv --policy examples/policy.json --output holdout-report.json

Provide an explicit, unique ID for each holdout prediction. Assign the ID to the underlying prediction before combining exports so duplicates can be detected. The bound assumes independent IID holdout units. validate uses the confidence level saved in the policy unless you pass --confidence-level. Empirical fitting reports observed risk and stores null in risk_upper_bound.

Exit code 0 means the holdout upper bound meets the declared risk limit. Exit code 1 means it does not, including when there are zero accepted samples. Exit code 2 indicates invalid input. Use an untouched holdout and interpret the result for its sampled distribution. See statistical scope.

Python API

from calibroute import PredictionRecord, fit_policy, route_batch

validation = [
    PredictionRecord("a", 0.98, True),
    PredictionRecord("b", 0.82, True),
    PredictionRecord("c", 0.55, False),
]

policy = fit_policy(validation, max_risk=0.10, min_coverage=0.50,
                    risk_method="empirical")  # Tiny illustrative sample only.
decisions, summary = route_batch(
    [PredictionRecord("new", 0.74)],
    policy,
)

Limitations

CalibRoute is an evaluation and routing tool, not a safety certification. Thresholds should be revalidated after changes to the model, task, prompt, or deployment distribution. See design principles for details.

Development

python -m unittest discover -s tests -v

See CONTRIBUTING.md and ROADMAP.md. Tried it on a task? Share a usage report, including unsuccessful trials or integration difficulties. Release preparation is documented in releasing. The Financial NER case study demonstrates the adapter and cross-domain failure pattern on 2,098 derived prediction rows. The ShiftGuard agent example audits 384 frozen proposals, keeps task templates separate, and reports when no confidence threshold meets the declared empirical risk target. Use CalibRoute 0.4.0 or newer for the convert-shiftguard command.

License

MIT. See LICENSE.

Metadata

Release files for calibroute-ai 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for calibroute-ai 0.5.0
File Size Uploaded Provenance Actions
calibroute_ai-0.5.0.tar.gz 361.8 kB With provenance Details for calibroute_ai-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for calibroute-ai 0.5.0
File Interpreter ABI Platform Actions
calibroute_ai-0.5.0-py3-none-any.whl Python 3 none any Details for calibroute_ai-0.5.0-py3-none-any.whl

Total release size: 388.0 kB

Release files / calibroute_ai-0.5.0.tar.gz

Download URL calibroute_ai-0.5.0.tar.gz
Size 361.8 kB
Tags Source
SHA-256 checksum
How to use checksums
7f36d28d7929d163f096297beb4581632cd127e2d545b2077b3fa2ca0f6138e8
BLAKE2b-256 checksum
How to use checksums
f15410d48bb96f2dfe4c5650349a72978db415cfb1676f72b5b4b3db9f0eae7b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / calibroute_ai-0.5.0-py3-none-any.whl

Download URL calibroute_ai-0.5.0-py3-none-any.whl
Size 26.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e9ae4c02736c608b7faf35406b12152951f97908b5556d5b45c84cee246abb7d
BLAKE2b-256 checksum
How to use checksums
7f7ce966114b18bc728b4c6cd8e20ee0226676e2b080ac8b8b336b8ffd624a23
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for calibroute-ai 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for calibroute_ai-0.5.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page