Skip to main content

TrustBench

ML Reliability, Robustness, Uncertainty & Fairness Evaluation Framework

PyPI version Python Versions License CI Status Documentation Tests


TrustBench is a framework-independent machine learning evaluation library designed to measure how reliable models remain under distribution shifts, input corruptions, calibration failures, and algorithmic bias.

Traditional model evaluation focuses almost exclusively on accuracy. TrustBench investigates whether models remain reliable in deployment:

  • Calibration: Does predicted confidence reflect the true probability of correctness?
  • Robustness: How gracefully does performance degrade under physical, digital, and environmental corruptions?
  • Uncertainty & Error Detection: Can predictive entropy and free energy identify incorrect predictions before downstream failures occur?
  • Selective Prediction: What error rate is achieved when models defer low-confidence decisions?
  • Multi-Framework: Evaluate PyTorch, Scikit-learn, ONNX Runtime, TensorFlow/Keras, and JAX models using identical contracts.
  • Multi-Task: Native evaluators for Classification, Regression, Information Retrieval, and Semantic Segmentation.

Architecture Overview

flowchart TD
    UserConfig["trustbench.yaml / Python API"] --> TB["TrustBench Orchestrator"]
    
    subgraph FrameworkAdapters ["Framework Adapters (Optional)"]
        Torch["PyTorch (nn.Module)"]
        SKLearn["Scikit-learn (Estimators)"]
        ONNX["ONNX Runtime (.onnx)"]
        Keras["TensorFlow / Keras"]
        JAX["JAX / Flax"]
    end
    
    subgraph TaskPipelines ["Task-Specific Evaluators"]
        Cls["Classification\n(Accuracy, Macro-F1, Confusion Matrix)"]
        Calib["Calibration & Reliability\n(ECE, AdaECE, Brier, Temperature)"]
        Unc["Uncertainty & Detection\n(Entropy, Energy, AUROC, AUPRC)"]
        Sel["Selective Prediction\n(Risk-Coverage, AURC)"]
        Rob["Robustness Suite\n(Noise, Blur, Brightness 1-5)"]
        Reg["Regression Suite\n(MAE, RMSE, R², 95% PICP)"]
        Ret["Retrieval Suite\n(Recall@K, MRR, nDCG@K)"]
        Seg["Segmentation Suite\n(mIoU, Dice, Pixel Acc)"]
        Fair["Fairness Suite\n(Demographic Parity, Disparate Impact)"]
    end
    
    subgraph StorageReporting ["Storage & Reporting"]
        FSStore["Filesystem Store\n(Atomic Writes & Checksums)"]
        SQLiteStore["SQLite & SQLAlchemy\n(Queryable Registry)"]
        HTMLRep["Interactive HTML Report\n(Zero-Dependency Plotly)"]
        JSONRep["Versioned JSON Schema\n(v1.0 Lineage)"]
    end

    FrameworkAdapters --> TB
    TB --> TaskPipelines
    TaskPipelines --> StorageReporting

Installation

TrustBench features a lightweight core with optional framework groups:

# Core installation (NumPy, SciPy, Scikit-learn, Click, Pydantic)
pip install trustbench

# For PyTorch model evaluation
pip install "trustbench[pytorch]"

# For ONNX Runtime inference
pip install "trustbench[onnx]"

# Full installation (all frameworks, storage, and visualization)
pip install "trustbench[all]"

See the Installation Guide for detailed platform requirements.


5-Minute Quickstart

1. Command Line Interface (CLI)

# Initialize a starter configuration
trustbench init

# Validate configuration paths and settings
trustbench validate trustbench.yaml

# Run evaluation and generate reports
trustbench evaluate trustbench.yaml

2. Python API

import numpy as np
from sklearn.linear_model import LogisticRegression
from trustbench import TrustBench, InMemoryDatasetAdapter

# 1. Train or load your model
X_train = np.random.randn(200, 4)
y_train = np.random.randint(0, 2, size=200)
clf = LogisticRegression().fit(X_train, y_train)

# 2. Prepare test dataset
X_test = np.random.randn(50, 4)
y_test = np.random.randint(0, 2, size=50)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

# 3. Evaluate reliability
bench = TrustBench(model=clf, dataset=dataset)
result = bench.evaluate()

print(f"Accuracy:    {result.metrics['classification']['accuracy']:.2%}")
print(f"Brier Score: {result.metrics['calibration']['brier_score']:.4f}")
print(f"ECE:         {result.metrics['calibration']['ece']:.4f}")

Multi-Task Evaluation

Continuous Regression Evaluation

from trustbench import evaluate_regression, InMemoryDatasetAdapter
from sklearn.linear_model import Ridge
import numpy as np

model = Ridge().fit(X_train, y_train)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)

result = evaluate_regression(model, dataset)
print(f"MAE:  {result.metrics['mae']:.3f}")
print(f"RMSE: {result.metrics['rmse']:.3f}")
print(f"R²:   {result.metrics['r2']:.3f}")

Information Retrieval Evaluation

from trustbench import evaluate_retrieval

ground_truth = {"q1": ["docA", "docB"], "q2": ["docC"]}
retrieved = {"q1": ["docA", "docX", "docB"], "q2": ["docY", "docC"]}

res = evaluate_retrieval(ground_truth, retrieved, k_list=(1, 3))
print(f"MRR:      {res.metrics['mrr']:.3f}")
print(f"Recall@1: {res.metrics['recall@1']:.3f}")
print(f"nDCG@3:   {res.metrics['ndcg@3']:.3f}")

Semantic Segmentation Evaluation

from trustbench import evaluate_segmentation

result = evaluate_segmentation(segmentation_model, segmentation_dataset, num_classes=21)
print(f"Mean IoU:       {result.metrics['miou']:.3f}")
print(f"Dice (F1):      {result.metrics['dice']:.3f}")
print(f"Pixel Accuracy: {result.metrics['pixel_accuracy']:.2%}")

Key Metrics Reference

Category Metrics Implemented
Classification Accuracy, Balanced Accuracy, Precision, Recall, Macro-F1, Confusion Matrix
Calibration Expected Calibration Error (ECE), Adaptive ECE, Maximum Calibration Error (MCE), Brier Score, NLL
Uncertainty Predictive Shannon Entropy, Maximum Softmax Probability (MSP), Free Energy
Error Detection AUROC, AUPRC, False Positive Rate at 95% TPR (FPR@95%TPR), Minimum Detection Error
Selective Prediction Risk-Coverage Trade-off Curves, Area Under Risk-Coverage (AURC), Excess AURC
Robustness Gaussian Noise, Blur, Brightness, Defocus, Contrast, Corrupted Accuracy Drop
Regression MAE, MSE, RMSE, $R^2$, MedAE, MAPE, Max Error, 95% Interval Coverage (PICP, MPIW)
Retrieval Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Recall@K, Precision@K, nDCG@K
Segmentation Mean IoU (mIoU), Dice (F1), Pixel Accuracy, Frequency-Weighted IoU (FWIoU)
Fairness Demographic Parity Difference, Disparate Impact Ratio, Equalized Odds Difference

See Metrics Documentation for mathematical derivations.


Tutorials & Examples

Ready-to-run examples are available in the examples/ directory:

  1. 01_evaluate_pytorch_model.py: Evaluate custom PyTorch nn.Module.
  2. 02_evaluate_cifar10.py: End-to-end CIFAR-10 ResNet evaluation via configuration.
  3. 03_robustness_benchmark.py: Evaluate accuracy degradation across noise & corruptions.
  4. 04_analyze_calibration.py: Compute ECE and fit validation temperature scaling.
  5. 05_evaluate_uncertainty.py: Predict error detection AUROC and risk-coverage curves.
  6. 06_compare_checkpoints.py: Quantify statistical differentials between model checkpoints.
  7. 07_save_and_reproduce_experiment.py: Atomic experiment persistence and checksum verification.
  8. 08_create_custom_metric.py: Register custom evaluation scoring functions.
  9. 09_create_custom_model_adapter.py: Implement the runtime ModelAdapter protocol.

Documentation

Full documentation is available on the Documentation Website or in the docs/ directory:


Contributing

We welcome contributions! Please see CONTRIBUTING.md and our Code of Conduct.

# Run tests
pytest

# Check code style
ruff check .

# Static type analysis
mypy src/trustbench/core src/trustbench/reporting src/trustbench/metrics

License

TrustBench is open-source software licensed under the Apache License, Version 2.0.

Metadata

Release files for trustbench 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trustbench 0.2.2
File Size Uploaded Provenance Actions
trustbench-0.2.2.tar.gz 120.1 kB With provenance Details for trustbench-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trustbench 0.2.2
File Interpreter ABI Platform Actions
trustbench-0.2.2-py3-none-any.whl Python 3 none any Details for trustbench-0.2.2-py3-none-any.whl

Total release size: 285.5 kB

Release files / trustbench-0.2.2.tar.gz

Download URL trustbench-0.2.2.tar.gz
Size 120.1 kB
Tags Source
SHA-256 checksum
How to use checksums
81beb3e43de04e481a2d6a7800c8891d8eafa105f02abec70a82bcf7a7c24cbb
BLAKE2b-256 checksum
How to use checksums
ace97cd369e237218c5bdfbf3c914b2c9c3c190d360ebaad3d8f49aaee2c4076
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / trustbench-0.2.2-py3-none-any.whl

Download URL trustbench-0.2.2-py3-none-any.whl
Size 165.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
737ed2b3f16dec71eec70134eb03bc60ed2ba0331a3b95b1e36598aa3db63f7c
BLAKE2b-256 checksum
How to use checksums
fe5633f543a40c7d8028ddd3e77f6da1fa6f988b4150b8e10e9c79c686eecd38
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for trustbench 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for trustbench-0.2.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page