Skip to main content

TrustBench

ML Reliability, Robustness, Uncertainty & Fairness Evaluation Framework

PyPI version Python Versions License CI Status Documentation


TrustBench is a framework-independent machine learning evaluation library designed to measure how reliable models remain under distribution shifts, input corruptions, calibration failures, and algorithmic bias.

Traditional model evaluation focuses almost exclusively on accuracy. TrustBench investigates whether models remain reliable in deployment:

  • Calibration: Does predicted confidence reflect the true probability of correctness?
  • Robustness: How gracefully does performance degrade under physical, digital, and environmental corruptions?
  • Uncertainty & Error Detection: Can predictive entropy and free energy identify incorrect predictions before downstream failures occur?
  • Selective Prediction: What error rate is achieved when models defer low-confidence decisions?
  • Multi-Framework: Evaluate PyTorch, Scikit-learn, ONNX Runtime, TensorFlow/Keras, and JAX models using identical contracts.
  • Multi-Task: Native evaluators for Classification, Regression, Information Retrieval, and Semantic Segmentation.

Architecture Overview

flowchart TD
    UserConfig["trustbench.yaml / Python API"] --> TB["TrustBench Orchestrator"]
    
    subgraph FrameworkAdapters ["Framework Adapters (Optional)"]
        Torch["PyTorch (nn.Module)"]
        SKLearn["Scikit-learn (Estimators)"]
        ONNX["ONNX Runtime (.onnx)"]
        Keras["TensorFlow / Keras"]
        JAX["JAX / Flax"]
    end
    
    subgraph TaskPipelines ["Task-Specific Evaluators"]
        Cls["Classification\n(Accuracy, Macro-F1, Confusion Matrix)"]
        Calib["Calibration & Reliability\n(ECE, AdaECE, Brier, Temperature)"]
        Unc["Uncertainty & Detection\n(Entropy, Energy, AUROC, AUPRC)"]
        Sel["Selective Prediction\n(Risk-Coverage, AURC)"]
        Rob["Robustness Suite\n(Noise, Blur, Brightness 1-5)"]
        Reg["Regression Suite\n(MAE, RMSE, R², 95% PICP)"]
        Ret["Retrieval Suite\n(Recall@K, MRR, nDCG@K)"]
        Seg["Segmentation Suite\n(mIoU, Dice, Pixel Acc)"]
        Fair["Fairness Suite\n(Demographic Parity, Disparate Impact)"]
    end
    
    subgraph StorageReporting ["Storage & Reporting"]
        FSStore["Filesystem Store\n(Atomic Writes & Checksums)"]
        SQLiteStore["SQLite & SQLAlchemy\n(Queryable Registry)"]
        HTMLRep["Interactive HTML Report\n(Zero-Dependency Plotly)"]
        JSONRep["Versioned JSON Schema\n(v1.0 Lineage)"]
    end

    FrameworkAdapters --> TB
    TB --> TaskPipelines
    TaskPipelines --> StorageReporting

Installation

TrustBench features a lightweight core with optional framework groups:

# Core installation (NumPy, SciPy, Scikit-learn, Click, Pydantic)
pip install trustbench

# For PyTorch model evaluation
pip install "trustbench[pytorch]"

# For ONNX Runtime inference
pip install "trustbench[onnx]"

# Full installation (all frameworks, storage, and visualization)
pip install "trustbench[all]"

See the Installation Guide for detailed platform requirements.


5-Minute Quickstart

1. Command Line Interface (CLI)

# Initialize a starter configuration
trustbench init

# Validate configuration paths and settings
trustbench validate trustbench.yaml

# Run evaluation and generate reports
trustbench evaluate trustbench.yaml

2. Python API

import numpy as np
from sklearn.linear_model import LogisticRegression
from trustbench import TrustBench, InMemoryDatasetAdapter

# 1. Train or load your model
X_train = np.random.randn(200, 4)
y_train = np.random.randint(0, 2, size=200)
clf = LogisticRegression().fit(X_train, y_train)

# 2. Prepare test dataset
X_test = np.random.randn(50, 4)
y_test = np.random.randint(0, 2, size=50)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

# 3. Evaluate reliability
bench = TrustBench(model=clf, dataset=dataset)
result = bench.evaluate()

print(f"Accuracy:    {result.metrics['classification']['accuracy']:.2%}")
print(f"Brier Score: {result.metrics['calibration']['brier_score']:.4f}")
print(f"ECE:         {result.metrics['calibration']['ece']:.4f}")

Multi-Task Evaluation

Continuous Regression Evaluation

from trustbench import evaluate_regression, InMemoryDatasetAdapter
from sklearn.linear_model import Ridge
import numpy as np

model = Ridge().fit(X_train, y_train)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

result = evaluate_regression(model, dataset)
print(f"MAE:  {result.metrics['mae']:.3f}")
print(f"RMSE: {result.metrics['rmse']:.3f}")
print(f"R²:   {result.metrics['r2']:.3f}")

Information Retrieval Evaluation

from trustbench import evaluate_retrieval

ground_truth = {"q1": ["docA", "docB"], "q2": ["docC"]}
retrieved = {"q1": ["docA", "docX", "docB"], "q2": ["docY", "docC"]}

res = evaluate_retrieval(ground_truth, retrieved, k_list=(1, 3))
print(f"MRR:      {res.metrics['mrr']:.3f}")
print(f"Recall@1: {res.metrics['recall@1']:.3f}")
print(f"nDCG@3:   {res.metrics['ndcg@3']:.3f}")

Semantic Segmentation Evaluation

from trustbench import evaluate_segmentation

result = evaluate_segmentation(segmentation_model, segmentation_dataset, num_classes=21)
print(f"Mean IoU:       {result.metrics['miou']:.3f}")
print(f"Dice (F1):      {result.metrics['dice']:.3f}")
print(f"Pixel Accuracy: {result.metrics['pixel_accuracy']:.2%}")

Key Metrics Reference

Category Metrics Implemented
Classification Accuracy, Balanced Accuracy, Precision, Recall, Macro-F1, Confusion Matrix
Calibration Expected Calibration Error (ECE), Adaptive ECE, Maximum Calibration Error (MCE), Brier Score, NLL
Uncertainty Predictive Shannon Entropy, Maximum Softmax Probability (MSP), Free Energy
Error Detection AUROC, AUPRC, False Positive Rate at 95% TPR (FPR@95%TPR), Minimum Detection Error
Selective Prediction Risk-Coverage Trade-off Curves, Area Under Risk-Coverage (AURC), Excess AURC
Robustness Gaussian Noise, Blur, Brightness, Defocus, Contrast, Corrupted Accuracy Drop
Regression MAE, MSE, RMSE, $R^2$, MedAE, MAPE, Max Error, 95% Interval Coverage (PICP, MPIW)
Retrieval Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Recall@K, Precision@K, nDCG@K
Segmentation Mean IoU (mIoU), Dice (F1), Pixel Accuracy, Frequency-Weighted IoU (FWIoU)
Fairness Demographic Parity Difference, Disparate Impact Ratio, Equalized Odds Difference

See Metrics Documentation for mathematical derivations.

Metric Verification & Standards Compliance

To guarantee mathematical correctness and numerical fidelity, TrustBench metrics are rigorously tested directly against established reference implementations and peer-reviewed literature benchmarks on identical inputs:

TrustBench Metric Domain Reference Implementation / Peer-Reviewed Standard Verification Test Suite
Classification (Acc, Balanced Acc, Macro-F1, Precision, Recall, Confusion Matrix) scikit-learn (sklearn.metrics) tests/unit/test_metric_reference_benchmarks.py
Regression (MAE, MSE, RMSE, $R^2$, MedAE, Max Error) scikit-learn (sklearn.metrics) tests/unit/test_metric_reference_benchmarks.py
Calibration (Brier Score, Negative Log-Likelihood) scikit-learn (brier_score_loss, log_loss) tests/unit/test_metric_reference_benchmarks.py
Calibration (Equal-width ECE, Adaptive Quantile ECE) Guo et al. (2017), Naeini et al. (2015), Ding et al. (2020) tests/unit/test_metric_reference_benchmarks.py
Error Detection (AUROC, AUPRC, FPR@95%TPR) scikit-learn (roc_auc_score, precision_recall_curve, auc) tests/unit/test_metric_reference_benchmarks.py
Fairness & Bias (Demographic Parity, Equalized Odds, Disparate Impact) Fairlearn Mathematical Specification tests/unit/test_metric_reference_benchmarks.py
Selective Prediction (Risk-Coverage, Optimal AURC, Excess AURC) Geifman & El-Yaniv (2017) Analytical Bounds tests/unit/test_metric_reference_benchmarks.py

Tutorials & Examples

Ready-to-run examples are available in the examples/ directory:

  1. 01_evaluate_pytorch_model.py: Evaluate custom PyTorch nn.Module.
  2. 02_evaluate_cifar10.py: End-to-end CIFAR-10 ResNet evaluation via configuration.
  3. 03_robustness_benchmark.py: Evaluate accuracy degradation across noise & corruptions.
  4. 04_analyze_calibration.py: Compute ECE and fit validation temperature scaling.
  5. 05_evaluate_uncertainty.py: Predict error detection AUROC and risk-coverage curves.
  6. 06_compare_checkpoints.py: Quantify statistical differentials between model checkpoints.
  7. 07_save_and_reproduce_experiment.py: Atomic experiment persistence and checksum verification.
  8. 08_create_custom_metric.py: Register custom evaluation scoring functions.
  9. 09_create_custom_model_adapter.py: Implement the runtime ModelAdapter protocol.

Documentation

Full documentation is available on the Documentation Website or in the docs/ directory:


Contributing

We welcome contributions! Please see CONTRIBUTING.md and our Code of Conduct.

# Run tests
pytest

# Check code style
ruff check .

# Static type analysis
mypy src/trustbench --ignore-missing-imports

License

TrustBench is open-source software licensed under the Apache License, Version 2.0.

Metadata

Release files for trustbench 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trustbench 0.2.3
File Size Uploaded Provenance Actions
trustbench-0.2.3.tar.gz 120.8 kB With provenance Details for trustbench-0.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trustbench 0.2.3
File Interpreter ABI Platform Actions
trustbench-0.2.3-py3-none-any.whl Python 3 none any Details for trustbench-0.2.3-py3-none-any.whl

Total release size: 286.8 kB

Release files / trustbench-0.2.3.tar.gz

Download URL trustbench-0.2.3.tar.gz
Size 120.8 kB
Tags Source
SHA-256 checksum
How to use checksums
038b2f88e9e8e9bc6914a329beb712eb17ec6dc309a6aba6a98711ad78e7d1c1
BLAKE2b-256 checksum
How to use checksums
93f5148b213335de4bf5f0f0d446b0b4f16cf3116e284ac4bc990fedd56c5a6e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / trustbench-0.2.3-py3-none-any.whl

Download URL trustbench-0.2.3-py3-none-any.whl
Size 166.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
926143cb5dd60f3def3d22e000348c150dbdf3185580418bd57000269ab20b98
BLAKE2b-256 checksum
How to use checksums
cbb751b6180eeff62fe32f5dc19f4b9a69038fb780fc3f3b87b087f9c1a67a07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for trustbench 0.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for trustbench-0.2.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page