Skip to main content

TrustBench

ML Reliability, Robustness, Uncertainty & Fairness Evaluation Framework

PyPI version Python Versions License CI Status Tests


TrustBench is a framework-independent machine learning evaluation library designed to measure how reliable models remain under distribution shifts, input corruptions, calibration failures, and algorithmic bias.

Traditional model evaluation focuses almost exclusively on accuracy. TrustBench investigates whether models remain reliable in deployment:

  • Calibration: Does predicted confidence reflect the true probability of correctness?
  • Robustness: How gracefully does performance degrade under physical, digital, and environmental corruptions?
  • Uncertainty & Error Detection: Can predictive entropy and free energy identify incorrect predictions before downstream failures occur?
  • Selective Prediction: What error rate is achieved when models defer low-confidence decisions?
  • Multi-Framework: Evaluate PyTorch, Scikit-learn, ONNX Runtime, TensorFlow/Keras, and JAX models using identical contracts.
  • Multi-Task: Native evaluators for Classification, Regression, Information Retrieval, and Semantic Segmentation.

Architecture Overview

flowchart TD
    UserConfig["trustbench.yaml / Python API"] --> TB["TrustBench Orchestrator"]
    
    subgraph FrameworkAdapters ["Framework Adapters (Optional)"]
        Torch["PyTorch (nn.Module)"]
        SKLearn["Scikit-learn (Estimators)"]
        ONNX["ONNX Runtime (.onnx)"]
        Keras["TensorFlow / Keras"]
        JAX["JAX / Flax"]
    end
    
    subgraph TaskPipelines ["Task-Specific Evaluators"]
        Cls["Classification\n(Accuracy, Macro-F1, Confusion Matrix)"]
        Calib["Calibration & Reliability\n(ECE, AdaECE, Brier, Temperature)"]
        Unc["Uncertainty & Detection\n(Entropy, Energy, AUROC, AUPRC)"]
        Sel["Selective Prediction\n(Risk-Coverage, AURC)"]
        Rob["Robustness Suite\n(Noise, Blur, Brightness 1-5)"]
        Reg["Regression Suite\n(MAE, RMSE, R², 95% PICP)"]
        Ret["Retrieval Suite\n(Recall@K, MRR, nDCG@K)"]
        Seg["Segmentation Suite\n(mIoU, Dice, Pixel Acc)"]
        Fair["Fairness Suite\n(Demographic Parity, Disparate Impact)"]
    end
    
    subgraph StorageReporting ["Storage & Reporting"]
        FSStore["Filesystem Store\n(Atomic Writes & Checksums)"]
        SQLiteStore["SQLite & SQLAlchemy\n(Queryable Registry)"]
        HTMLRep["Interactive HTML Report\n(Zero-Dependency Plotly)"]
        JSONRep["Versioned JSON Schema\n(v1.0 Lineage)"]
    end

    FrameworkAdapters --> TB
    TB --> TaskPipelines
    TaskPipelines --> StorageReporting

Installation

TrustBench features a lightweight core with optional framework groups:

# Core installation (NumPy, SciPy, Scikit-learn, Click, Pydantic)
pip install trustbench

# For PyTorch model evaluation
pip install "trustbench[pytorch]"

# For ONNX Runtime inference
pip install "trustbench[onnx]"

# Full installation (all frameworks, storage, and visualization)
pip install "trustbench[all]"

See the Installation Guide for detailed platform requirements.


5-Minute Quickstart

1. Command Line Interface (CLI)

# Initialize a starter configuration
trustbench init

# Validate configuration paths and settings
trustbench validate trustbench.yaml

# Run evaluation and generate reports
trustbench evaluate trustbench.yaml

2. Python API

import numpy as np
from sklearn.linear_model import LogisticRegression
from trustbench import TrustBench, InMemoryDatasetAdapter

# 1. Train or load your model
X_train = np.random.randn(200, 4)
y_train = np.random.randint(0, 2, size=200)
clf = LogisticRegression().fit(X_train, y_train)

# 2. Prepare test dataset
X_test = np.random.randn(50, 4)
y_test = np.random.randint(0, 2, size=50)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

# 3. Evaluate reliability
bench = TrustBench(model=clf, dataset=dataset)
result = bench.evaluate()

print(f"Accuracy:    {result.metrics['classification']['accuracy']:.2%}")
print(f"Brier Score: {result.metrics['calibration']['brier_score']:.4f}")
print(f"ECE:         {result.metrics['calibration']['ece']:.4f}")

Multi-Task Evaluation

Continuous Regression Evaluation

from trustbench import evaluate_regression, InMemoryDatasetAdapter
from sklearn.linear_model import Ridge
import numpy as np

model = Ridge().fit(X_train, y_train)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

result = evaluate_regression(model, dataset)
print(f"MAE:  {result.metrics['mae']:.3f}")
print(f"RMSE: {result.metrics['rmse']:.3f}")
print(f"R²:   {result.metrics['r2']:.3f}")

Information Retrieval Evaluation

from trustbench import evaluate_retrieval

ground_truth = {"q1": ["docA", "docB"], "q2": ["docC"]}
retrieved = {"q1": ["docA", "docX", "docB"], "q2": ["docY", "docC"]}

res = evaluate_retrieval(ground_truth, retrieved, k_list=(1, 3))
print(f"MRR:      {res.metrics['mrr']:.3f}")
print(f"Recall@1: {res.metrics['recall@1']:.3f}")
print(f"nDCG@3:   {res.metrics['ndcg@3']:.3f}")

Semantic Segmentation Evaluation

from trustbench import evaluate_segmentation

result = evaluate_segmentation(segmentation_model, segmentation_dataset, num_classes=21)
print(f"Mean IoU:       {result.metrics['miou']:.3f}")
print(f"Dice (F1):      {result.metrics['dice']:.3f}")
print(f"Pixel Accuracy: {result.metrics['pixel_accuracy']:.2%}")

Key Metrics Reference

Category Metrics Implemented
Classification Accuracy, Balanced Accuracy, Precision, Recall, Macro-F1, Confusion Matrix
Calibration Expected Calibration Error (ECE), Adaptive ECE, Maximum Calibration Error (MCE), Brier Score, NLL
Uncertainty Predictive Shannon Entropy, Maximum Softmax Probability (MSP), Free Energy
Error Detection AUROC, AUPRC, False Positive Rate at 95% TPR (FPR@95%TPR), Minimum Detection Error
Selective Prediction Risk-Coverage Trade-off Curves, Area Under Risk-Coverage (AURC), Excess AURC
Robustness Gaussian Noise, Blur, Brightness, Defocus, Contrast, Corrupted Accuracy Drop
Regression MAE, MSE, RMSE, $R^2$, MedAE, MAPE, Max Error, 95% Interval Coverage (PICP, MPIW)
Retrieval Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Recall@K, Precision@K, nDCG@K
Segmentation Mean IoU (mIoU), Dice (F1), Pixel Accuracy, Frequency-Weighted IoU (FWIoU)
Fairness Demographic Parity Difference, Disparate Impact Ratio, Equalized Odds Difference

See Metrics Documentation for mathematical derivations.


Tutorials & Examples

Ready-to-run examples are available in the examples/ directory:

  1. 01_evaluate_pytorch_model.py: Evaluate custom PyTorch nn.Module.
  2. 02_evaluate_cifar10.py: End-to-end CIFAR-10 ResNet evaluation via configuration.
  3. 03_robustness_benchmark.py: Evaluate accuracy degradation across noise & corruptions.
  4. 04_analyze_calibration.py: Compute ECE and fit validation temperature scaling.
  5. 05_evaluate_uncertainty.py: Predict error detection AUROC and risk-coverage curves.
  6. 06_compare_checkpoints.py: Quantify statistical differentials between model checkpoints.
  7. 07_save_and_reproduce_experiment.py: Atomic experiment persistence and checksum verification.
  8. 08_create_custom_metric.py: Register custom evaluation scoring functions.
  9. 09_create_custom_model_adapter.py: Implement the runtime ModelAdapter protocol.

Documentation

Full documentation is available in the docs/ directory:


Contributing

We welcome contributions! Please see CONTRIBUTING.md and our Code of Conduct.

# Run tests
pytest

# Check code style
ruff check .

# Static type analysis
mypy src/trustbench/core src/trustbench/reporting src/trustbench/metrics

License

TrustBench is open-source software licensed under the Apache License, Version 2.0.

Metadata

Release files for trustbench 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trustbench 0.2.0
File Size Uploaded Provenance Actions
trustbench-0.2.0.tar.gz 119.6 kB With provenance Details for trustbench-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trustbench 0.2.0
File Interpreter ABI Platform Actions
trustbench-0.2.0-py3-none-any.whl Python 3 none any Details for trustbench-0.2.0-py3-none-any.whl

Total release size: 284.6 kB

Release files / trustbench-0.2.0.tar.gz

Download URL trustbench-0.2.0.tar.gz
Size 119.6 kB
Tags Source
SHA-256 checksum
How to use checksums
a5771cd81344ab2a06625b5e5ba75605c0b6c46af45eae704e5e4a5aa6347e5d
BLAKE2b-256 checksum
How to use checksums
c2a780bc2445b6c8343a3aa7786f64e5ccaeba9eb38c4c80be595ab8c6782e26
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Release files / trustbench-0.2.0-py3-none-any.whl

Download URL trustbench-0.2.0-py3-none-any.whl
Size 165.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
217e19ddbe49d3f857149afacecc5cd514c368f0d70f7757216f03d844208a65
BLAKE2b-256 checksum
How to use checksums
01b24ae32781049dd74a731cbe5f2fed05f56683566c08e60d587feabca6dee8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Security for trustbench 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for trustbench-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page