bencheval
Benchmark evaluation for tabular machine learning: turns a table of per-task results into a
leaderboard with Elo, win-rates, average ranks, improvability, and bootstrap confidence intervals.
It is the leaderboard engine behind TabArena and is developed in the
TabArena repository, but it does not depend on the
tabarena package, so any benchmark that produces (method, task, metric_error) rows can use it.
Experimental. The API is still moving. Pin the version you evaluate with; every
tabarenarelease pins the matchingbenchevalrelease.
Install
pip install --pre bencheval # metrics + leaderboards
pip install --pre "bencheval[plot]" # + matplotlib / seaborn / plotly for the plotting mixin
Releases on PyPI are pre-releases for now, so --pre (or uv pip install --prerelease=allow) is
needed. The git checkout is the recommended install for development; see the
TabArena README.
Usage
import pandas as pd
from bencheval.evaluator import BenchmarkEvaluator
# One row per (method, task[, seed]); lower `metric_error` is better.
data = pd.DataFrame(
{
"method": ["A", "B", "A", "B"],
"task": ["t1", "t1", "t2", "t2"],
"seed": [0, 0, 0, 0],
"metric_error": [0.10, 0.12, 0.30, 0.25],
"time_train_s": [1.0, 2.0, 1.5, 2.5],
"time_infer_s": [0.1, 0.2, 0.1, 0.2],
}
)
evaluator = BenchmarkEvaluator(seed_column="seed")
leaderboard = evaluator.leaderboard(data, include_error=True)
print(leaderboard)
examples/plots/run_generate_custom_leaderboard.py in the TabArena repository shows custom
leaderboard metrics, Elo calibration against a reference method, and seed averaging.
License
Apache-2.0. See the LICENSE file shipped with the distribution.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file bencheval-0.1.0.tar.gz.
File metadata
- Download URL: bencheval-0.1.0.tar.gz
- Upload date:
- Size: 55.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9c150bc291ac1f64b1cad4e957bc76875b75f796ac5dec4ba252d6ba39c37030
|
|
| MD5 |
b0219de83109ddf477327322d006811e
|
|
| BLAKE2b-256 |
f83e8179d95fce17188f6468a0c74d6722872277bf42e8686abc6cda4daa3a61
|
File details
Details for the file bencheval-0.1.0-py3-none-any.whl.
File metadata
- Download URL: bencheval-0.1.0-py3-none-any.whl
- Upload date:
- Size: 57.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
52b6e13c1329bf75c67cf3d1a2e38e008b7673275f69cc05982249b73af66230
|
|
| MD5 |
195fec188e6359440b7a45feac2f0c23
|
|
| BLAKE2b-256 |
a41929cfc48bf38f264f6a5a9520d9c47c19b4004c63352570adee19006df2d6
|
