Skip to main content

TrustTab

TrustTab is a Python framework for measuring predictive performance and probability reliability of tabular classifiers under controlled training-label noise and test-time feature shifts. It provides reproducible experiments and research-oriented summaries; it does not claim that any model is universally robust.

Capabilities

  • Logistic Regression and Random Forest, with optional XGBoost and TabPFN
  • German Credit and processed Cleveland Heart Disease loaders; Adult Census Income remains available as an additional dataset
  • Clean baselines, symmetric training-label noise, and numeric/categorical feature shifts
  • Opt-in Platt calibration experiments using separate clean calibration data
  • Accuracy, balanced accuracy, macro F1, ROC-AUC, Brier score, ECE, adaptive ECE, maximum calibration error, and log loss
  • Platt scaling in the integrated calibration experiment; isolated utilities also expose isotonic calibration
  • Paired statistical comparisons with multiple-comparison corrections
  • CSV benchmarks, Markdown reports, headless figures, and a local run registry

The package is organized under src/trusttab/: data and models prepare inputs and estimators; noise creates perturbations; experiments orchestrates runs and tracking; evaluation, statistics, visualization, and reporting analyze results.

Install

Python 3.11 or 3.12 is supported.

After the approved PyPI release, install the public distribution with:

python -m pip install trusttab

The package has not been published to PyPI yet. To install the current source checkout, use:

python -m pip install .

For development, tests, linting, and builds:

python -m pip install -e ".[dev]"

Optional models can be installed separately:

python -m pip install -e ".[xgboost]"
python -m pip install -e ".[tabpfn]"

The tabpfn extra pins TabPFN 10.0.0; it is not installed with core TrustTab dependencies. See the TabPFN compatibility and hardware notes before enabling it. TabPFN-3.5 weights are separately licensed and gated; users must accept the model terms and authorize their own local download. The default TabPFN-3.5 weights are under Prior Labs' non-commercial, non-production license; commercial or production use requires rights from Prior Labs. TrustTab does not bundle or redistribute model weights. See the TabPFN license and access notes. The current license also restricts use of model outputs to non-commercial purposes and includes user eligibility conditions; review the full Prior Labs license before accessing the checkpoint.

Platt calibration rescue

Calibration is an opt-in secondary experiment. See the calibration-rescue protocol and run its bounded one-model smoke configuration with:

python -m trusttab.experiments.run --config configs/calibration_rescue_smoke.yaml

The calibration experiment uses separate stratified train, calibration, and test subsets. Only training labels are noise-corrupted; calibration labels remain clean, and test labels are used only for final evaluation. Raw metrics remain in the standard columns and corresponding calibrated metrics use a calibrated_ prefix. Calibration is disabled in the core configuration by default, consistent with the blueprint's secondary-experiment design.

Quick start

From a repository checkout, run the small two-model, two-seed end-to-end benchmark (including label noise, feature shift, reporting, and local tracking):

trusttab --config configs/final_smoke_test.yaml

The YAML templates are repository files and are not bundled in the wheel. A pip-only installation can use the Python API below or provide its own YAML configuration; the CLI does not silently select the 200-condition research configuration.

The Python API also supports a unified benchmark:

from trusttab.api import run_robustness_benchmark

results = run_robustness_benchmark(
    dataset_name="german_credit",
    label_noise_levels=[0.0, 0.2],
    feature_shifts=[
        {"type": "mean_shift", "features": ["age"], "magnitude": 0.5}
    ],
    model_names=["logistic_regression", "random_forest"],
    seeds=[1, 2, 3],
    output_path="results/benchmark.csv",
)

Dataset loading may require network access on the first use if the source data is not already available. Unit tests use local fixtures and do not download datasets.

See the public API guide for supported imports and examples/quickstart.py for a runnable synthetic example that does not download a dataset.

Datasets and attribution

The research benchmark pair is German Credit and the processed Cleveland subset of UCI Heart Disease (UCI dataset ID 45). TrustTab reads only processed.cleveland.data; it does not combine the Cleveland, Hungarian, Switzerland, and VA databases. The original num target is preserved in y.attrs["original_num"] and mapped to binary labels as 0 → 0 and 1, 2, 3, 4 → 1. The 303 rows and 13 source features are retained. Source missing values in ca and thal remain missing in the returned features; the experiment pipeline imputes them after splitting, using training data only. An optional data_home argument caches the processed Cleveland rows, including the original num target, as a local CSV.

Heart Disease citation: Janosi, A., Steinbrunn, W., Pfisterer, M., & Detrano, R. (1989). Heart Disease. UCI Machine Learning Repository, dataset ID 45, DOI 10.24432/C52P4X. The dataset is licensed CC BY 4.0. Attribution: Heart Disease dataset by Janosi, Steinbrunn, Pfisterer, and Detrano, hosted by the UCI Machine Learning Repository, used under CC BY 4.0. See dataset loading details for source and missing-value behavior.

Configuration and CLI

YAML configurations are validated before execution. See configs/a6_smoke_test.yaml for a small runnable configuration and configs/experiments.yaml for the configured 200-condition research matrix. Use configs/phase3_smoke_test.yaml for a small two-model, two-noise-level check. Dataset downloads may require network access when the data is not cached.

trusttab --help
trusttab --config configs/experiments.yaml
trusttab --list-runs --runs-dir experiments/runs
trusttab --inspect-run RUN_ID --runs-dir experiments/runs
trusttab --compare-runs RUN_ID_A RUN_ID_B --runs-dir experiments/runs
trusttab --verify-run RUN_ID --runs-dir experiments/runs

Enable local tracking with tracking: {enabled: true} in YAML. Run commands write benchmark output to the configured output location; tracked artifacts are stored under experiments/runs unless configured otherwise.

Analysis and reports

The reporting API accepts unified benchmark results:

from trusttab.reporting import generate_robustness_report

artifacts = generate_robustness_report(results, "results/report")

It produces a Markdown report, CSV summaries, and headless PNG figures. See the benchmark, metrics, statistics, visualization, reporting, and tracking guides.

The report also provides seed-blocked Friedman/Nemenyi model comparisons, paired raw-versus-calibrated ECE/Brier tests, normalized degradation slopes, effect sizes, and small-sample confidence intervals. These are exploratory with five seeds, and the two datasets are analyzed separately rather than treated as a sample of datasets. See docs/statistics.md for the analysis protocol and its limits. Running the report does not run experiments or establish empirical conclusions by itself.

Outputs and reproducibility

Benchmark CSV rows identify the dataset, model, scenario, perturbation, corruption level, seed, metrics, sample counts, and baseline link. Tracked runs also retain original and normalized configuration, configuration hash, environment and Git metadata, dataset fingerprints, results, and report artifacts. Re-execution checks numerical agreement using configurable absolute and relative tolerances. Numerical reproducibility does not imply bitwise identity across hardware, dependency versions, or nondeterministic estimators.

Completed core research run

The completed core experiment evaluates whether predictive performance and probability reliability change differently as training labels become noisier. It covers Logistic Regression, Random Forest, XGBoost, and TabPFN on German Credit (OpenML ID 31) and the processed Cleveland subset of UCI Heart Disease (UCI ID 45), at noise rates 0%, 10%, 20%, 30%, and 40%, with seeds 0 through 4. This is 200 noise conditions plus 40 clean-baseline reference rows, for 240 result rows. Adult remains an optional loader and is not part of this core matrix. Cleveland uses only the processed Cleveland file, not a merge of UCI's four heart-disease databases.

Training labels use independent per-example Bernoulli flips; the realized corruption count varies around its expectation. Calibration and test labels remain clean. The implementation follows the blueprint's mathematical definition of independent flips; an earlier fixed-count pseudocode fragment was inconsistent and was corrected. The original research pair of German Credit and Heart Disease is retained, with the approved processed Cleveland subset as the Heart Disease source.

The run records Accuracy, Balanced Accuracy, Macro-F1, ROC-AUC, ECE, and Brier Score. The primary model inference is dataset-specific, seed-blocked Friedman testing with Nemenyi post-hoc comparisons gated on the omnibus result and Holm correction within the planned families. Degradation-slope comparisons use paired seed-level results. Descriptive intervals bootstrap the five seed units. The generated compatibility table also includes supplementary paired t-tests; these are exploratory, not the primary planned analysis. Calibration was disabled in the core matrix, so no core raw-versus-calibrated inference is available and RQ4 (the Platt calibration question) remains unanswered.

Interpretation is limited by five seeds and two fixed datasets; these do not support broad claims about all datasets. Cleveland's 61-row test partition makes its metric estimates especially coarse. The saved aggregate results do not retain per-observation predictions, so final metrics cannot be independently recomputed from the CSV alone. The preserved run, manifest, report tables, figures, and RQ0-RQ4 findings are in the local working copy at experiments/runs/trusttab-core-20261114/; they are research outputs, not package data, and are not included in the wheel.

Research assumptions and limitations

  • Label noise uses independent Bernoulli flips on training labels: each label is flipped with the configured probability. The realized flip count is random (expected to be noise_rate × n_train), not fixed to that value. Test labels remain clean. Real annotation errors may be class-dependent or structured.
  • Feature shifts are controlled synthetic interventions and do not represent every real distribution shift.
  • ECE and related binned calibration metrics depend on binning choices and can be noisy on small samples. ROC-AUC is undefined when an evaluation split has one class.
  • Paired tests require aligned observations/seeds and rely on their own assumptions. Small seed counts have limited power; multiplicity correction does not remove this limitation.
  • German Credit, processed Cleveland Heart Disease, and Adult are limited benchmark datasets and should not be treated as representative of all tabular applications.
  • Optional model implementations may have model-specific constraints and nondeterminism. Reproducibility metadata helps interpret reruns but cannot guarantee identical results everywhere.
  • Robustness under a specified perturbation is distinct from clean predictive accuracy and does not establish universal reliability.

Citation

For the software, cite the TrustTab repository and released version used in your work: Shreya Nipunge et al., TrustTab: Reliability-Aware Stress Testing for Tabular Machine Learning, GitHub repository https://github.com/Shreya-nipunge/trusttab, version 0.1.0 (update the version to the release actually used). For the Cleveland benchmark, cite the UCI Heart Disease dataset separately using the DOI and CC BY 4.0 attribution above. Do not cite smoke-test outputs as research findings.

Development

python -m pip install -e ".[dev]"
pytest -q
ruff check src tests
ruff format --check src tests
python -m build

Run these checks locally on Python 3.11 or 3.12. The test suite uses local fixtures and does not download datasets or require optional model packages.

Ruff format currently excludes 29 legacy source and test files from its format check; the remaining files are checked. These exclusions are an explicit cleanup backlog, not a claim that those files match formatter output.

Metadata

Release files for trusttab 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trusttab 0.1.0
File Size Uploaded Provenance Actions
trusttab-0.1.0.tar.gz 128.9 kB Without provenance Details for trusttab-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trusttab 0.1.0
File Interpreter ABI Platform Actions
trusttab-0.1.0-py3-none-any.whl Python 3 none any Details for trusttab-0.1.0-py3-none-any.whl

Total release size: 213.1 kB

Release files / trusttab-0.1.0.tar.gz

Download URL trusttab-0.1.0.tar.gz
Size 128.9 kB
Tags Source
SHA-256 checksum
How to use checksums
b1c1f792309092560b5ad9f5950dba490afd9083269180ae62cdc68df076daa6
BLAKE2b-256 checksum
How to use checksums
0ad0462c51a652539876018002807fdf9546c3e4a774e7c3fac2e1aa0da3427c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Release files / trusttab-0.1.0-py3-none-any.whl

Download URL trusttab-0.1.0-py3-none-any.whl
Size 84.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b8f3bf0f38339c4ec7fe8c30d6122dba5bb393ff218fe98bedda2bdba1e4742f
BLAKE2b-256 checksum
How to use checksums
19a971e6928ba61edd1a922a8e938cc3b2301ec6a5baa172ba306740fd88714a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Security for trusttab 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page