TrustBench is a framework-independent machine learning evaluation library designed to measure how reliable models remain under distribution shifts, input corruptions, calibration failures, and algorithmic bias.
Traditional model evaluation focuses almost exclusively on accuracy. TrustBench investigates whether models remain reliable in deployment:
- Calibration: Does predicted confidence reflect the true probability of correctness?
- Robustness: How gracefully does performance degrade under physical, digital, and environmental corruptions?
- Uncertainty & Error Detection: Can predictive entropy and free energy identify incorrect predictions before downstream failures occur?
- Selective Prediction: What error rate is achieved when models defer low-confidence decisions?
- Multi-Framework: Evaluate PyTorch, Scikit-learn, ONNX Runtime, TensorFlow/Keras, and JAX models using identical contracts.
- Multi-Task: Native evaluators for Classification, Regression, Information Retrieval, and Semantic Segmentation.
Architecture Overview
flowchart TD
UserConfig["trustbench.yaml / Python API"] --> TB["TrustBench Orchestrator"]
subgraph FrameworkAdapters ["Framework Adapters (Optional)"]
Torch["PyTorch (nn.Module)"]
SKLearn["Scikit-learn (Estimators)"]
ONNX["ONNX Runtime (.onnx)"]
Keras["TensorFlow / Keras"]
JAX["JAX / Flax"]
end
subgraph TaskPipelines ["Task-Specific Evaluators"]
Cls["Classification\n(Accuracy, Macro-F1, Confusion Matrix)"]
Calib["Calibration & Reliability\n(ECE, AdaECE, Brier, Temperature)"]
Unc["Uncertainty & Detection\n(Entropy, Energy, AUROC, AUPRC)"]
Sel["Selective Prediction\n(Risk-Coverage, AURC)"]
Rob["Robustness Suite\n(Noise, Blur, Brightness 1-5)"]
Reg["Regression Suite\n(MAE, RMSE, R², 95% PICP)"]
Ret["Retrieval Suite\n(Recall@K, MRR, nDCG@K)"]
Seg["Segmentation Suite\n(mIoU, Dice, Pixel Acc)"]
Fair["Fairness Suite\n(Demographic Parity, Disparate Impact)"]
end
subgraph StorageReporting ["Storage & Reporting"]
FSStore["Filesystem Store\n(Atomic Writes & Checksums)"]
SQLiteStore["SQLite & SQLAlchemy\n(Queryable Registry)"]
HTMLRep["Interactive HTML Report\n(Zero-Dependency Plotly)"]
JSONRep["Versioned JSON Schema\n(v1.0 Lineage)"]
end
FrameworkAdapters --> TB
TB --> TaskPipelines
TaskPipelines --> StorageReporting
Installation
TrustBench features a lightweight core with optional framework groups:
# Core installation (NumPy, SciPy, Scikit-learn, Click, Pydantic)
pip install trustbench
# For PyTorch model evaluation
pip install "trustbench[pytorch]"
# For ONNX Runtime inference
pip install "trustbench[onnx]"
# Full installation (all frameworks, storage, and visualization)
pip install "trustbench[all]"
See the Installation Guide for detailed platform requirements.
5-Minute Quickstart
1. Command Line Interface (CLI)
# Initialize a starter configuration
trustbench init
# Validate configuration paths and settings
trustbench validate trustbench.yaml
# Run evaluation and generate reports
trustbench evaluate trustbench.yaml
2. Python API
import numpy as np
from sklearn.linear_model import LogisticRegression
from trustbench import TrustBench, InMemoryDatasetAdapter
# 1. Train or load your model
X_train = np.random.randn(200, 4)
y_train = np.random.randint(0, 2, size=200)
clf = LogisticRegression().fit(X_train, y_train)
# 2. Prepare test dataset
X_test = np.random.randn(50, 4)
y_test = np.random.randint(0, 2, size=50)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)
# 3. Evaluate reliability
bench = TrustBench(model=clf, dataset=dataset)
result = bench.evaluate()
print(f"Accuracy: {result.metrics['classification']['accuracy']:.2%}")
print(f"Brier Score: {result.metrics['calibration']['brier_score']:.4f}")
print(f"ECE: {result.metrics['calibration']['ece']:.4f}")
Multi-Task Evaluation
Continuous Regression Evaluation
from trustbench import evaluate_regression, InMemoryDatasetAdapter
from sklearn.linear_model import Ridge
import numpy as np
model = Ridge().fit(X_train, y_train)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)
result = evaluate_regression(model, dataset)
print(f"MAE: {result.metrics['mae']:.3f}")
print(f"RMSE: {result.metrics['rmse']:.3f}")
print(f"R²: {result.metrics['r2']:.3f}")
Information Retrieval Evaluation
from trustbench import evaluate_retrieval
ground_truth = {"q1": ["docA", "docB"], "q2": ["docC"]}
retrieved = {"q1": ["docA", "docX", "docB"], "q2": ["docY", "docC"]}
res = evaluate_retrieval(ground_truth, retrieved, k_list=(1, 3))
print(f"MRR: {res.metrics['mrr']:.3f}")
print(f"Recall@1: {res.metrics['recall@1']:.3f}")
print(f"nDCG@3: {res.metrics['ndcg@3']:.3f}")
Semantic Segmentation Evaluation
from trustbench import evaluate_segmentation
result = evaluate_segmentation(segmentation_model, segmentation_dataset, num_classes=21)
print(f"Mean IoU: {result.metrics['miou']:.3f}")
print(f"Dice (F1): {result.metrics['dice']:.3f}")
print(f"Pixel Accuracy: {result.metrics['pixel_accuracy']:.2%}")
Key Metrics Reference
| Category | Metrics Implemented |
|---|---|
| Classification | Accuracy, Balanced Accuracy, Precision, Recall, Macro-F1, Confusion Matrix |
| Calibration | Expected Calibration Error (ECE), Adaptive ECE, Maximum Calibration Error (MCE), Brier Score, NLL |
| Uncertainty | Predictive Shannon Entropy, Maximum Softmax Probability (MSP), Free Energy |
| Error Detection | AUROC, AUPRC, False Positive Rate at 95% TPR (FPR@95%TPR), Minimum Detection Error |
| Selective Prediction | Risk-Coverage Trade-off Curves, Area Under Risk-Coverage (AURC), Excess AURC |
| Robustness | Gaussian Noise, Blur, Brightness, Defocus, Contrast, Corrupted Accuracy Drop |
| Regression | MAE, MSE, RMSE, $R^2$, MedAE, MAPE, Max Error, 95% Interval Coverage (PICP, MPIW) |
| Retrieval | Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Recall@K, Precision@K, nDCG@K |
| Segmentation | Mean IoU (mIoU), Dice (F1), Pixel Accuracy, Frequency-Weighted IoU (FWIoU) |
| Fairness | Demographic Parity Difference, Disparate Impact Ratio, Equalized Odds Difference |
See Metrics Documentation for mathematical derivations.
Metric Verification & Standards Compliance
To guarantee mathematical correctness and numerical fidelity, TrustBench metrics are rigorously tested directly against established reference implementations and peer-reviewed literature benchmarks on identical inputs:
| TrustBench Metric Domain | Reference Implementation / Peer-Reviewed Standard | Verification Test Suite |
|---|---|---|
| Classification (Acc, Balanced Acc, Macro-F1, Precision, Recall, Confusion Matrix) | scikit-learn (sklearn.metrics) |
tests/unit/test_metric_reference_benchmarks.py |
| Regression (MAE, MSE, RMSE, $R^2$, MedAE, Max Error) | scikit-learn (sklearn.metrics) |
tests/unit/test_metric_reference_benchmarks.py |
| Calibration (Brier Score, Negative Log-Likelihood) | scikit-learn (brier_score_loss, log_loss) |
tests/unit/test_metric_reference_benchmarks.py |
| Calibration (Equal-width ECE, Adaptive Quantile ECE) | Guo et al. (2017), Naeini et al. (2015), Ding et al. (2020) | tests/unit/test_metric_reference_benchmarks.py |
| Error Detection (AUROC, AUPRC, FPR@95%TPR) | scikit-learn (roc_auc_score, precision_recall_curve, auc) |
tests/unit/test_metric_reference_benchmarks.py |
| Fairness & Bias (Demographic Parity, Equalized Odds, Disparate Impact) | Fairlearn Mathematical Specification | tests/unit/test_metric_reference_benchmarks.py |
| Selective Prediction (Risk-Coverage, Optimal AURC, Excess AURC) | Geifman & El-Yaniv (2017) Analytical Bounds | tests/unit/test_metric_reference_benchmarks.py |
Tutorials & Examples
Ready-to-run examples are available in the examples/ directory:
- 01_evaluate_pytorch_model.py: Evaluate custom PyTorch
nn.Module. - 02_evaluate_cifar10.py: End-to-end CIFAR-10 ResNet evaluation via configuration.
- 03_robustness_benchmark.py: Evaluate accuracy degradation across noise & corruptions.
- 04_analyze_calibration.py: Compute ECE and fit validation temperature scaling.
- 05_evaluate_uncertainty.py: Predict error detection AUROC and risk-coverage curves.
- 06_compare_checkpoints.py: Quantify statistical differentials between model checkpoints.
- 07_save_and_reproduce_experiment.py: Atomic experiment persistence and checksum verification.
- 08_create_custom_metric.py: Register custom evaluation scoring functions.
- 09_create_custom_model_adapter.py: Implement the runtime
ModelAdapterprotocol.
Documentation
Full documentation is available on the Documentation Website or in the docs/ directory:
- Installation Guide
- Quickstart Guide
- Python API Reference
- CLI Reference
- Configuration Guide
- Model Adapters
- Dataset Adapters
- Metrics Documentation
- Robustness Testing
- Experiment Tracking
- Reporting & Visualization
- Plugin Development
- Troubleshooting & FAQ
Contributing
We welcome contributions! Please see CONTRIBUTING.md and our Code of Conduct.
# Run tests
pytest
# Check code style
ruff check .
# Static type analysis
mypy src/trustbench --ignore-missing-imports
License
TrustBench is open-source software licensed under the Apache License, Version 2.0.
Metadata
Release files for trustbench 0.2.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| trustbench-0.2.3.tar.gz | 120.8 kB | With provenance | Details for trustbench-0.2.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| trustbench-0.2.3-py3-none-any.whl | Python 3 | none | any | Details for trustbench-0.2.3-py3-none-any.whl |
Total release size: 286.8 kB
Release files / trustbench-0.2.3.tar.gz
| Download URL | trustbench-0.2.3.tar.gz |
|---|---|
| Size | 120.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
038b2f88e9e8e9bc6914a329beb712eb17ec6dc309a6aba6a98711ad78e7d1c1
|
|
BLAKE2b-256 checksum How to use checksums |
93f5148b213335de4bf5f0f0d446b0b4f16cf3116e284ac4bc990fedd56c5a6e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logRelease files / trustbench-0.2.3-py3-none-any.whl
| Download URL | trustbench-0.2.3-py3-none-any.whl |
|---|---|
| Size | 166.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
926143cb5dd60f3def3d22e000348c150dbdf3185580418bd57000269ab20b98
|
|
BLAKE2b-256 checksum How to use checksums |
cbb751b6180eeff62fe32f5dc19f4b9a69038fb780fc3f3b87b087f9c1a67a07
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logSecurity for trustbench 0.2.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for trustbench-0.2.3.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.