🏆 Leaderboard · 📄 Paper · 🤗 Datasets · 📚 Collection · 🚀 Getting started · ➕ Add a model · 📝 Cite

MulTaBench is a benchmark for multimodal tabular learning: tables whose columns hold free text or images alongside numbers and categories. Its 40 core datasets, 20 text-tabular and 20 image-tabular, were kept only if they show both joint signal, each modality carrying signal about the target that the others don't, and target awareness, the best representation of the text or images depending on the task. 40 more datasets with joint signal are released alongside them, 80 in total.

Models are ranked on the live leaderboard, but the benchmark's purpose is to compare ways of encoding text and images for tabular learners, and new methods are welcome.

Getting started

The core package loads the datasets, their splits and cached embeddings, without torch or any model package:

pip install multabench

from multabench import Size, list_datasets, load_embeddings, load_split, score

datasets = list_datasets(modality="text", task="regression", tier="core")

split = load_split("MUL_IMAGE_PETFINDER", fold=0, size=Size.TEN_K)
split.x_train, split.y_train, split.x_test, split.y_test

embeddings = load_embeddings("MUL_TEXT_WINE_REVIEW", fold=0, size=Size.TEN_K, encoder="e5-small")
embeddings.train["description"], embeddings.test["description"]

predictions = model.fit(split.x_train, split.y_train).predict(split.x_test)
result = score(split.task_type, y_true=split.y_test, y_pred=predictions)
result.test_error

load_published_results() returns every leaderboard run.

The loaders download from the Hugging Face Hub at pinned revisions. To read a local copy instead, e.g. offline or from a read-only mount, pass data_dir, holding one folder per Hub repo named without the org, with images extracted: <data_dir>/<dataset repo>/data.parquet and <data_dir>/embeddings-e5-small/<dataset repo>/embeddings.npz. data_dir_files(dataset, encoders) lists the paths one dataset needs there.

split = load_split("MUL_IMAGE_PETFINDER", fold=0, data_dir="/data/multabench")

with_image_files=False loads an image dataset's table without its image files, enough for the cached image embeddings, which are looked up by file name; the split's image_folder is then None.

To run the benchmark itself, set up the environment and evaluate one model on one fold:

source init.sh && source .venv/bin/activate
cp .env.example .env  # Hugging Face token

python benchmark.py --model light --dataset_name MUL_IMAGE_PETFINDER --fold 0

Running the benchmark covers sweeps, hardware and collecting results; Cached embeddings, Datasets and Protocol, including how Elo is computed and how MulTaBench differs from TabArena, cover the rest.

Adding a model

Open a pull request that adds the model; once it is merged, we run it from master on the leaderboard's hardware, so every result comes from the same machines and from code anyone can check out. The pull request adds:

1. A wrapper, multabench/baselines/<model>.py: a TabularModel subclass, e.g. multabench/baselines/tabdpt13.py.
2. The model in BASELINES in benchmark.py.
3. Its dependencies as an extra in pyproject.toml, locked in requirements.txt, or in its own requirements-<ENV>.txt if they conflict.
4. A row in space/data/models.csv, with a small logo in space/assets/logos/ and a link to its paper.

Use the model's default hyperparameters, pin anything it downloads, and say on which datasets it can't run, e.g. too many classes. Running the benchmark has the details.

Citation

@article{arazi2026multabench,
  title   = {MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image},
  author  = {Arazi, Alan and Shapira, Eilam and Grunblat, Shoham and Ventura, Mor and Hoffer, Elad and Blayer, Gioia and
             Holzm{\"u}ller, David and Purucker, Lennart and Varoquaux, Ga{\"e}l and Hutter, Frank and Reichart, Roi},
  journal = {arXiv preprint arXiv:2605.10616},
  year    = {2026}
}

License

The code is under the MIT license; each dataset keeps the terms of its source, linked in MULTABENCH_SOURCES in multabench/datasets/all_datasets.py.