BenchHub
Benchmark model predictions against curated datasets — with a typed contract end-to-end. Pick (or import) a dataset, define metrics and visualizations in Python, submit predictions with the client, and see how your model ranks sample-by-sample. Live at https://runbenchhub.com.
What makes it different
Everything hangs off one idea — a typed contract. Every field (a dataset
column, a leaderboard input, a prediction) declares a kind
(image, mask, depth, audio, label, label_list, bboxes,
scalar, text, json, sequence, coco_detections, plus any
user-registered kind). The dataset, the benchhub-client, and the metric
engine all agree on kinds, so data is decoded, scored, and rendered without
guessing. Kinds are defined in benchhub/types.py and listed live at
/supported_types.
Features
- Passwordless auth — GitHub, Google, or a one-time email code.
- Self-service HuggingFace import — one button: the tabular importer
(Croissant / parquet) falls back automatically to a file-tree mapper
(paired files, packed
.npz/.h5/archives, video clips) with a "describe the structure" role wizard, loaders for file/token/npz/json/csv/parquet/hdf5/zip/tar/gz/sequence, a decode preview, variant fan-out, draft autosave, and a split/subfolder picker for huge repos. Datasets shown before import via their HF card. - Two-tier storage — datasets cache as a cheap preview tier; each leaderboard materializes a chosen sample subset at full resolution.
- Metrics & visualizations in Python — typed signatures, per-sample + aggregated, pooling (mean/median/percentile/min/max), dependency chaining.
- Hardened sandbox — all user-supplied code (metrics, visualizations,
and a registered type's
visualize) runs in a short-lived--network=none --read-only, memory/CPU-capped container — never in-process on the server. - User-registered data types — declare a new
kind(its storage + a sandboxedvisualize(blob, params)) from the web (/supported_types) or the client; it joins the global kind namespace and renders in the views. benchhub-client+ dev kit —iter_samples→predict→submit; programmatic dataset creation;create_metric/create_visualization/create_datatype; andbenchhub.author.test_metric/test_visualizationto iterate locally before uploading.- Per-row visibility (
public/unlisted/private) with dependency guards — once another user depends on your dataset/LB (binds it / submits to it), it can't be made private or deleted. - Split-bucket quotas — 50 GB public + 10 GB private per user (live usage on Account settings + the Storage-usage page).
- Async processing with Celery (Redis broker); API tokens; public landing,
catalog (
/leaderboards,/datasets), and profiles (/u/<id>).
Quickstart (submitting to a leaderboard)
pip install -U benchhub-client # PyPI: benchhub-client
import benchhub as bh
client = bh.Client(token="bh_...") # token from /settings/api_tokens
sub = client.submission(LB_ID, name="my-model-v1")
for sample_name, inputs in client.iter_samples(LB_ID):
image = inputs["image"].array # decoded bh.Image -> (H,W,3) uint8
pred = my_model(image)
sub.predict(sample_name, label_pred=bh.Label(int(pred)))
print(sub.submit(description="ResNet-50"))
Authoring metrics / visualizations / data types
import benchhub as bh
def my_iou(gt: bh.Mask, pred: bh.Mask): # input_kinds auto-derive from annotations
g, p = gt.array, pred.array
inter = ((g == 1) & (p == 1)).sum(); union = ((g == 1) | (p == 1)).sum()
return float(inter / union) if union else 1.0
bh.author.test_metric(my_iou, gt=gt_mask, pred=pred_mask) # iterate locally
client.create_metric("my_iou", my_iou) # then upload (sandboxed server-side)
client.create_visualization(...) (returns a PIL.Image) and
client.create_datatype(...) (a new kind + a sandboxed visualize) work the
same way.
Documentation
- In-app docs at
/docs: overview, core concepts, importing data, data types, leaderboards, writing metrics & visualizations, submitting predictions, API/client reference, tutorials. - Architecture:
docs/ARCHITECTURE.md(editable drawio source underdocs/diagrams/). - Dev notes / history:
CLAUDE.md(durable architecture + gotchas), the granular subsystem notes underskills/, and the dated notes underdocs/. - Feature requests + bugs: GitHub issues.
Run it locally
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
redis-server # 1. broker (port 6379)
celery -A app.celery worker --loglevel=info # 2. worker
python app.py # 3. web app -> http://localhost:6060
The database + uploads live outside the repo in a data directory — set its
location with BENCHHUB_DATA_DIR=/some/path. To run user code in the
container sandbox locally, install Docker, build the runner image
(docker build -f runner/Dockerfile -t benchhub-runner .), and set
BENCHHUB_SANDBOX_METRICS=1.
Tests
pytest tests/ # ~1080 tests; use tests/ (not bare pytest)
Deployment
Self-hosted on an Ubuntu box at runbenchhub.com (gunicorn + celery + redis
under systemd, nginx + certbot, Cloudflare DNS-only). The operational
runbook — push flow, .env keys, logs, rollback — is
docs/SELFHOST_RUNBOOK.md.
License
MIT — see LICENSE.
Metadata
Release files for benchhub-client 0.1.11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| benchhub_client-0.1.11.tar.gz | 308.5 kB | Without provenance | Details for benchhub_client-0.1.11.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| benchhub_client-0.1.11-py3-none-any.whl | Python 3 | none | any | Details for benchhub_client-0.1.11-py3-none-any.whl |
Total release size: 425.0 kB
Release files / benchhub_client-0.1.11.tar.gz
| Download URL | benchhub_client-0.1.11.tar.gz |
|---|---|
| Size | 308.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
663ba15da6177d828d50db7ed620671a595a968e69c6b6a169b9f533cc19b81b
|
|
BLAKE2b-256 checksum How to use checksums |
5d0cd5d2f5f09edea3b83fa7889ef0595cb59c765b2ed33a83b02386b30ac3e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Release files / benchhub_client-0.1.11-py3-none-any.whl
| Download URL | benchhub_client-0.1.11-py3-none-any.whl |
|---|---|
| Size | 116.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3227474f9c9bf3a0d60fa2959ff564fca81e16c32f9152f6a805e54f095d0dcc
|
|
BLAKE2b-256 checksum How to use checksums |
e80f5f578fe90fafc54d8f3ba5c6375fab6e6c6224692db4307940e980975dbc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Security for benchhub-client 0.1.11
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.