relbench 3.0.0

News · Get Started · Tutorials · Leaderboard · BYOD · Citations

News

RelBench 3 loads every dataset family below from the Hugging Face Hub with one call — relbench.load_dataset("<org>/<repo>/<name>") — the core databases (stanford-star/relbench-v1, stanford-star/relbench-v2-extra), CTU/ReDeLEx (stanford-star/redelex), 4DBInfer (stanford-star/dbinfer) and TGB (stanford-star/tgb); the [ctu] extra is gone. MIMIC-IV requires PhysioNet credentials and is not hosted. TGB and 4DBInfer tasks that use their own scoring protocols load as data only (see MIGRATION.md).

• Aug 2026 — RelBench v3 released: datasets and tasks load straight from Hugging Face, a new leaderboard with automated submissions, and bug fixes (migration guide).
• Jun 2026 — Datasets, models, and community hub migrated to Hugging Face.
• Mar 2026 — RelBench v2 paper accepted at the ICLR 2026 DATA-FM workshop.
• Feb 2026 — Temporal Graph Benchmark integration: time-stamped event streams as relational schemas.
• Jan 2026 — RelBench v2 released: four new databases (SALT, RateBeer, arXiv, MIMIC-IV), 36 new tasks, and a new Autocomplete task type; plus 70+ CTU datasets via ReDeLEx and 7 from 4DBInfer.
• Jan 2026 — ReDeLEx integration: 70+ CTU relational databases via relbench[ctu].
• Sep 2024 — RelBench paper accepted at NeurIPS 2024 Datasets & Benchmarks Track.
• Jul 2024 — RelBench released: the first open benchmark for predictive ML on relational databases.
• May 2024 — Relational Deep Learning position paper accepted at ICML 2024.
• Nov 2023 — Relational Deep Learning introduced in a keynote by Jure Leskovec at the LoG Conference (slides).

Get Started

pip install relbench             # data + task loading
pip install "relbench[example]"  # + PyTorch Geometric & PyTorch Frame, for the GNN examples
pip install pyg-lib -f https://data.pyg.org/whl/torch-2.9.0+cpu.html  # neighbor sampling; use the index matching your torch/CUDA build

Load a dataset and a task — both come straight from the Hub, with no per-dataset code:

import relbench

dataset = relbench.load_dataset("rel-f1")   # or a HuggingFace 'org/repo[/subdir]', or a local path
db = dataset.get_db()                  # rows after test_timestamp are hidden

task = dataset.load_task("driver-position")   # dataset.get_task_names() lists them
train_table = task.get_table("train")  # train / val / test label tables
test_table  = task.get_table("test")   # the target column is hidden on test

# ... train any model on db + train_table, predict on the test entities ...
metrics = task.evaluate(test_pred)     # standard metric for the task type

dataset.val_timestamp / dataset.test_timestamp give the temporal split points. RelBench is framework-agnostic — bring any modeling stack. For a reference Graph Neural Network on PyTorch Geometric + PyTorch Frame, see relbench.modeling and the runnable scripts in examples/.

Tutorials

Open these directly in Google Colab — no setup required:

Tutorial	What it covers
Quickstart	Load a dataset/task, explore the schema, run a baseline
Training a GNN	A GNN baseline for an entity task (PyG + PyTorch Frame)

Leaderboard

The RelBench leaderboard ranks methods by their test-set performance, averaged over a fixed task set. There are three independent boards — classification (12 tasks), regression (9), and recommendation (10); the task lists are in relbench.submit.LEADERBOARD_TASKS. You can submit to any of them; each requires predictions for all of its tasks.

To submit:

1. Write one prediction CSV per task, named <dataset>__<task>.csv, into a directory:

   relbench.submit.write_prediction_table(task, test_pred, "preds/rel-f1__driver-position.csv")
   

2. Validate and package the directory — this scores every CSV against the test tables, prints a verdict per leaderboard, and writes clean submission zip file(s):

   python -m relbench.submit preds/
   

3. Open a submission issue on this repository: fill in the short form and upload the zip file(s) into it.

The submission is validated automatically and the report is posted on the issue; once a maintainer approves, your entry appears on the leaderboard.

BYOD (Bring Your Own Data)

You can easily express your own databases and tasks in the RelBench format. A dataset is a self-describing folder — a manifest.yaml (tables, keys, the foreign-key graph, the time splits), one plain parquet per table, and a tasks/ subdirectory — that you publish to the Hugging Face Hub. RelBench loads it straight from its org/repo[/subdir] address; there is no central registry to register with.

byod/README.md is the full walkthrough, and the published stanford-star/relbench-v1/rel-f1 is a complete worked example. For how RelBench's own databases were built, cleaned, and verified from their original sources, see provenance/.

If you upload your data in RelBench format to Hugging Face, please let us know by opening an issue / making a PR so we can list it here!

Citations

If you use RelBench, please cite the benchmark papers:

@inproceedings{relbench,
  title={RelBench: A Benchmark for Deep Learning on Relational Databases},
  author={Robinson, Joshua and Ranjan, Rishabh and Hu, Weihua and Huang, Kexin and Han, Jiaqi and Dobles, Alejandro and Fey, Matthias and Lenssen, Jan Eric and Yuan, Yiwen and Zhang, Zecheng and He, Xinwei and Leskovec, Jure},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}

@misc{relbenchv2,
  title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
  author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
  year={2026},
  eprint={2602.12606},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2602.12606}
}

Datasets sourced from external repositories (CTU/ReDeLEx, 4DBInfer, TGB) carry their own citations on their Hugging Face dataset cards.