TabBench-Bio

Website · GitHub · Codeberg

TabBench-Bio benchmarks machine learning on biological tables with many features and few samples, such as gene-expression and methylation data. It downloads datasets, fixes the train/test splits, fits models and ranks their predictions using fold-level Elo. The full configuration covers 33 classification and 13 regression datasets.

This repository contains the package and experiment definitions. The published website is built from Codeberg's pages branch.

Install

Use Python 3.13:

uv venv --python 3.13
source .venv/bin/activate
uv pip install tabbench-bio

For development, use uv pip install -e . from a source checkout. Dataset-loading dependencies are included by default. Missing datasets download automatically. Custom estimators need no AutoGluon or CUDA.

Benchmark your model

Define create_model(task) in my_model.py, returning a cloneable scikit-learn classifier or regressor. See the SVM example.

tabbench-bio evaluate --model my_model.py --task overall

This checks the latest published reference, reuses its verified local cache or downloads the current database, and uses its frozen folds. It writes my_model_results/report/report.html, Elo plots and CSV metrics. See Benchmark my model for options.

Datasets and models

The dataset registry gives each task a stable bio_id. Edit it to add datasets from a supported source, or set TABBENCH_BIO_DATASETS to your own registry file. A run config selects tasks through datasets_classification and datasets_regression. Default grids use null for both selections, expanding the enabled, curated tasks in this registry by task type. There are no separate default dataset rosters to maintain. Explicit lists still select subsets, and each run freezes its resolved dataset IDs.

Missing datasets download automatically and are cached. Curated embedding tables come from checksum-verified Parquet assets in the GitHub Release. Kaggle requires ~/.kaggle/kaggle.json. Use --cache-dir to select the cache or TABBENCH_BIO_LOCAL_DIR for local tables. The contributor guide explains how to add and check a dataset.

The model roster includes linear models, trees, neural networks and tabular foundation models. DUMMY provides a constant baseline, and Random Forest anchors Elo at 1000. New adapters need a ModelSpec in src/tabbench_bio/model_registry.py, including their environment profile.

Merge results and build the website

Combine a new model run with an existing database:

uv run --no-sync tabbench-bio merge results.sqlite results/my_model/results.sqlite \
  --output results/combined/results.sqlite

The output path must be new. The merge checks cell settings and held-out targets, removes duplicate attempts and leaves both inputs unchanged. It also accepts result directories containing writer databases.

The website/ submodule tracks Codeberg's pages branch. Initialise and update it before building from the repository root:

git submodule update --init website
git -C website switch pages
git -C website pull --ff-only
uv run --no-sync tabbench-bio leaderboard results/combined/results.sqlite --workers 4
python -m http.server 8000 --directory website

Open http://localhost:8000/. The build writes the dashboard, model cards, dataset pages and JSON data to website/. Per-cell PNG plots and CSV tables go in the Git-ignored website/local/ directory. It generates no per-cell HTML reports or SVGs. Use --out preview to build elsewhere.

The input database stays read-only. Building needs no raw datasets, GPU or LaTeX, and publishes nothing automatically. See the model guide for publishing and cache settings. The released database and SHA-256 checksum are listed in the artifact browser.

Read a leaderboard

Load a published SQLite bundle in Python:

from tabbench_bio import Leaderboard

sqlite_path = "results.sqlite"
print(Leaderboard.sqlite_cells(sqlite_path))
leaderboard = Leaderboard.from_sqlite(sqlite_path, cell="cap_10000_n100")
print(leaderboard.rank())

Or print one cell and export its plot and table:

uv run --no-sync tabbench-bio leaderboard --sqlite results.sqlite \
  --cell cap_10000_n100 --plot --csv leaderboard.csv

rank(), the terminal summary and plots use Bradley-Terry Elo from matching cross-validation folds. Each fold has weight 1 / target_fold_count, and the 95% intervals resample whole targets. The default is 2,000 bootstrap rounds, matching the paper. Normalised Score and fold means are descriptive summaries.

Strict results use recorded DUMMY metrics for failed fits and omit incomplete model-target fold sets. Elo needs RF and at least one comparable model. The website also provides adaptive and conditional views, explained in the model guide. Use --task to select classification or regression.

Caches

Dataset caches hold source downloads, assembled tables and prepared splits. Set cache_dir in a run config or TABBENCH_BIO_CACHE for the biological dataset cache.

Website builds cache fold metrics in <out>.cache/fold_metrics.sqlite and Elo in data/dashboard.json. Reuse the same output directory to avoid repeating work. Changed inputs, calculation code or relevant settings invalidate cached results. Completed metric batches survive an interrupted build. Elo is saved after a full build. Keep the metric cache locally. Deleting it forces recalculation.

--workers controls both metric and Elo calculations. Progress bars show completed folds and comparison pools, elapsed time and estimated time remaining.

Tuned models

Define a tuned variant in a JSON model roster: choose a supported parent adapter, give the variant its own key, and specify the candidate parameters. Selection uses training data only, with search and refitting sharing the run's time budget.

tabbench-bio MY-MODEL-TUNED --model-config my_grid.json --output results/my-tuned-model

See Add a tuned model for the JSON template and Run on a cluster for batch execution.

Licence

EUPL-1.2. See LICENSE. The vendored TabArena Elo helper retains Apache-2.0, as recorded in NOTICE and its licence. Datasets and models retain their providers' licences.

Paper citation

Kreuer, J., Ouaari, S., Hellmig, J., Braitinger, J., and Pfeifer, N. (2026). TabBench-Bio: A Living Benchmark for Machine Learning on High-Dimensional Biomedical Tables. arXiv:2609.07441. DOI: 10.48550/arXiv.2609.07441.

Agent skill for model selection

The model-selection skill helps an agent compare published results for a dataset's modality and feature/sample budgets, including uncertainty, failures, cost and training-data overlap. Install it in a compatible agent with:

npx skills add https://tabbench-bio.eu/skill.md

This downloads the Markdown skill. A browsing agent can also read the hosted version directly.

Dataset cache consistency

Raw dataset caches record a SHA-256 fingerprint of data-affecting registry fields and the loader version. A mismatch, including an older cache without a fingerprint, stops loading. Review the registry change, then explicitly refresh that dataset with load_bio_dataset(bio_id, cache_dir=..., force_refetch=True). Processed split cache paths include the same fingerprints in per-dataset directories. split_params.json records shared split settings; each directory has its own dataset_spec.json. Raw cache files are replaced atomically. Increment LOADER_CACHE_VERSION in bio/fingerprint.py when loader changes alter matrices, labels or biological groups.

Check a checkout

Run tabbench-bio doctor --cache-dir .cache to check registered adapter files, environment profiles, skill licences/copies, and existing raw and processed dataset caches. Add --site-dir website to verify generated skill assets. Add --model RF (repeatable) to test adapter importability in that model's installed environment; without it, only adapter file presence is checked. The command does not download data, fit models, or repair caches. Failures exit nonzero.

Legacy caches can be adopted explicitly after verifying that their source and task match the current registry, without downloading live sources again:

tabbench-bio cache-adopt --cache-dir .cache --dataset OpenML-1138 --reason "Verified against the original registry" --manifest results/run/split_manifest.json

The optional manifest verifies frozen target fingerprints before any cache is changed. Adoption records the reason and timestamp in cache metadata and refuses already-versioned or obsolete grouped caches. It cannot establish feature provenance; verify that separately. Raw fingerprints cover data-affecting fields, not descriptions or licences. Processed split paths include the individual dataset fingerprint, so filtered runs reuse full-run splits.