bencheval

Benchmark evaluation for tabular machine learning: turns a table of per-task results into a leaderboard with Elo, win-rates, average ranks, improvability, and bootstrap confidence intervals. It is the leaderboard engine behind TabArena and is developed in the TabArena repository, but it does not depend on the tabarena package, so any benchmark that produces (method, task, metric_error) rows can use it.

Experimental. The API is still moving. Pin the version you evaluate with; every tabarena release pins the matching bencheval release.

Install

pip install --pre bencheval          # metrics + leaderboards
pip install --pre "bencheval[plot]"  # + matplotlib / seaborn / plotly for the plotting mixin

Releases on PyPI are pre-releases for now, so --pre (or uv pip install --prerelease=allow) is needed. The git checkout is the recommended install for development; see the TabArena README.

Usage

import pandas as pd
from bencheval.evaluator import BenchmarkEvaluator

# One row per (method, task[, seed]); lower `metric_error` is better.
data = pd.DataFrame(
    {
        "method": ["A", "B", "A", "B"],
        "task": ["t1", "t1", "t2", "t2"],
        "seed": [0, 0, 0, 0],
        "metric_error": [0.10, 0.12, 0.30, 0.25],
        "time_train_s": [1.0, 2.0, 1.5, 2.5],
        "time_infer_s": [0.1, 0.2, 0.1, 0.2],
    }
)

evaluator = BenchmarkEvaluator(seed_column="seed")
leaderboard = evaluator.leaderboard(data, include_error=True)
print(leaderboard)

examples/plots/run_generate_custom_leaderboard.py in the TabArena repository shows custom leaderboard metrics, Elo calibration against a reference method, and seed averaging.

License

Apache-2.0. See the LICENSE file shipped with the distribution.

Metadata

Release files for bencheval 0.1.1.dev20260928100206

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bencheval 0.1.1.dev20260928100206
File	Size	Uploaded
bencheval-0.1.1.dev20260928100206.tar.gz	55.4 kB	Sep 28, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for bencheval 0.1.1.dev20260928100206
File	Interpreter	ABI	Platform	Reset
bencheval-0.1.1.dev20260928100206-py3-none-any.whl 57.8 kB Sep 28, 2026	Python 3	none	any	Details

Total release size: 113.3 kB

Release files / bencheval-0.1.1.dev20260928100206.tar.gz

Download URL	bencheval-0.1.1.dev20260928100206.tar.gz
Size	55.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`9cb166b782b90afc92788510d6b1358311800d2071fe90b9996b2313a05a67f2`
BLAKE2b-256 checksum How to use checksums	`986d76ee89aa143669bef7ca070605605b5373fd044fac042ccb0aef240b59ef`
Upload date	Sep 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / bencheval-0.1.1.dev20260928100206-py3-none-any.whl

Download URL	bencheval-0.1.1.dev20260928100206-py3-none-any.whl
Size	57.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`7a516b88e3086c67fae66c243af19d69da5b91bb777426cb14328851fc19af3c`
BLAKE2b-256 checksum How to use checksums	`d6f85bf558ff3e7b1eeb8618796ed26f2539b51edfbe9072601168c3b576afb9`
Upload date	Sep 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`