evaluma

A small Python package for comparing machine learning models across benchmark suites. Given a CSV of per-model, per-dataset scores, evaluma computes eight complementary views of the results:

• Aggregate ranking — point-estimate ranking via trimmed mean, mean, or median
• IQM ranking — interquartile mean with bootstrapped confidence intervals, following Agarwal et al. (2021)
• ELO ranking — MLE ELO ratings from pairwise head-to-head battles with bootstrap CIs and a win-rate matrix, following Erickson et al. (2025)
• Improvability ranking — mean percent error reduction each model needs to match the per-dataset best, in raw error space, following TabArena / BeyondArena
• Bayesian pairwise comparison — posterior probabilities that model A beats model B (or is practically equivalent), via baycomp
• Frequentist comparison — Friedman + Nemenyi (all-pairs) or Wilcoxon + Holm (reference model), following Demšar (2006)
• Dolan-Moré performance profiles — cumulative distribution of performance ratios and area-under-profile scores, following Dolan & Moré (2002)
• Rank sensitivity — Kendall τ-b with bootstrap CI measuring whether rankings hold across two experimental conditions, following Kendall (1945)

Documentation

Full documentation, including tutorials and API reference, is available at evaluma.readthedocs.io.

Installation

pip install evaluma

For a development install from source:

git clone https://github.com/nilsleh/evaluma
cd evaluma
pip install -e ".[dev]"

Quick start

Python API

import evaluma

bench = evaluma.load_df(
    "results.csv",
    model="model",
    dataset="dataset",
    metric="metric",
    score="score",
)

# Point-estimate aggregate ranking (trimmed mean)
agg = bench.aggregate_ranking()
print(agg.table)

# IQM ranking with 95% bootstrap CI (requires seed column)
iqm = bench.iqm_ranking()
print(iqm.table)
fig = iqm.plot()
fig.savefig("iqm.png")

# ELO ranking with win-rate matrix
elo = bench.elo_ranking()
print(elo.table)
fig = elo.plot_winrate()

# Improvability ranking (mean % error reduction needed to match the best)
imp = bench.improvability_ranking()
print(imp.table)
fig = imp.plot()

# Bayesian pairwise probabilities
bayes = bench.bayesian_comparison()
print(bayes.table)

# Frequentist comparison (Friedman + Nemenyi)
freq = bench.frequentist_comparison()
print(freq.table)

# Dolan-Moré performance profiles
profiles = bench.performance_profiles()
fig = profiles.plot()

# Rank sensitivity across two conditions
bench_b = evaluma.load_df("results_b.csv", model="model", dataset="dataset",
                          metric="metric", score="score")
sens = bench.rank_sensitivity(bench_b, cond_a="condA", cond_b="condB")
print(f"Kendall τ = {sens.tau:.3f}, 95% CI = {sens.tau_ci}")

CLI

# Run aggregate, Bayesian, frequentist, and performance-profile analyses
evaluma report results.csv \
    --model model --dataset dataset --metric metric --score score \
    --output results/

# Individual subcommands
evaluma aggregate   results.csv --model model --dataset dataset --metric metric --score score --output results/
evaluma rank        results.csv --model model --dataset dataset --metric metric --score score --seed seed --output results/
evaluma compare     results.csv --model model --dataset dataset --metric metric --score score --output results/
evaluma frequentist results.csv --model model --dataset dataset --metric metric --score score --output results/
evaluma profiles    results.csv --model model --dataset dataset --metric metric --score score --output results/

Each subcommand writes a .csv table and a .png figure to --output. ELO ranking and rank sensitivity are available via the Python API only.

Column mapping

If your CSV uses different column names, pass them explicitly:

evaluma report results.csv \
    --model experiment --dataset task --metric measure --score value \
    --output results/

Or put them in a YAML config file:

# config.yaml
model: experiment
dataset: task
metric: measure
score: value

evaluma report results.csv --config config.yaml --output results/

Lower-is-better metrics

bench = evaluma.load_df(
    "results.csv",
    model="model", dataset="dataset", metric="metric", score="score",
    metric_direction={"rmse": "min"},
)

evaluma report results.csv ... --metric-direction rmse:min

Filtering models or datasets

bench_ab = bench.select_models(["ModelA", "ModelB"])
bench_core = bench.select_datasets(["dataset1", "dataset2", "dataset3"])

Input format

evaluma expects a long-format CSV with one row per (model, dataset) combination:

model,dataset,metric,score
ModelA,dataset1,acc,0.91
ModelA,dataset2,acc,0.87
ModelB,dataset1,acc,0.84
...

Multiple seeds are supported — pass --seed seed_col and evaluma aggregates by mean before analysis.

Contributing

git clone https://github.com/nilsleh/evaluma
cd evaluma
pip install -e ".[dev]"

# Run tests
pytest --cov=evaluma --cov-report=term-missing

# Lint and format
ruff check .
ruff format .

# Type checking
ty check

Bug reports and pull requests are welcome on GitHub.

License

Apache License 2.0. See LICENSE for the full text.

Citation

If you use evaluma in your research, please cite:

@software{lehmann2026evaluma,
  author  = {Lehmann, Nils},
  title   = {evaluma: ML Benchmark Ranking Tools},
  year    = {2026},
  url     = {https://github.com/nilsleh/evaluma},
  version = {0.1.0},
}

also cite the works of the underlying methods and frameworks used:

@inproceedings{agarwal2021deep,
  title     = {Deep Reinforcement Learning at the Edge of the Statistical Precipice},
  author    = {Agarwal, Rishabh and Schwarzer, Max and Castro, Pablo Samuel
               and Courville, Aaron and Bellemare, Marc G.},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2021},
}

@inproceedings{erickson2025tabarena,
  title     = {{TabArena}: A Living Benchmark for Machine Learning on Tabular Data},
  author    = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej
               and Holzm{\"u}ller, David and Mutalik Desai, Prabhant
               and Salinas, David and Hutter, Frank},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2025},
}

@article{benavoli2017time,
  title   = {Time for a Change: a Tutorial for Comparing Multiple Classifiers
             Through Bayesian Analysis},
  author  = {Benavoli, Alessio and Corani, Giorgio and Dem{\v{s}}ar, Janez
             and Zaffalon, Marco},
  journal = {Journal of Machine Learning Research},
  volume  = {18},
  number  = {77},
  pages   = {1--36},
  year    = {2017},
}

@article{demsar2006statistical,
  title   = {Statistical Comparisons of Classifiers over Multiple Data Sets},
  author  = {Dem{\v{s}}ar, Janez},
  journal = {Journal of Machine Learning Research},
  volume  = {7},
  pages   = {1--30},
  year    = {2006},
}

@article{dolan2002benchmarking,
  title   = {Benchmarking Optimization Software with Performance Profiles},
  author  = {Dolan, Elizabeth D. and Mor{\'e}, Jorge J.},
  journal = {Mathematical Programming},
  volume  = {91},
  pages   = {201--213},
  year    = {2002},
}

@article{kendall1945treatment,
  title   = {The Treatment of Ties in Ranking Problems},
  author  = {Kendall, Maurice G.},
  journal = {Biometrika},
  volume  = {33},
  number  = {3},
  pages   = {239--251},
  year    = {1945},
  doi     = {10.1093/biomet/33.3.239},
}