The package ships a toy CuratedContainer so you can poke at the full API — schema, dtypes, splits, describe() — without touching the network. Identical interface to a downloaded BeyondArena container.

from data_foundry.curation_container import CuratedContainer
from data_foundry.examples import get_toy_container_path

container = CuratedContainer.load(get_toy_container_path())
print(container.describe())          # full identity + dtypes + task + splits
print(container.dataset.shape)       # the actual DataFrame
print(container.task_metadata.split_regime)  # "iid", "temporal_non_iid", or "grouped_non_iid"

Full inspection script (every metadata field printed): examples/load_curated_container.py.

Download a single BeyondArena container by name (or UUID) and iterate its outer CV splits. The collection resolves the container against your local cache; subsequent runs hit disk, not the network.

from data_foundry.collections import BEYOND_ARENA

container = BEYOND_ARENA.get_dataset("airfoil_self_noise")
df = container.dataset
target = container.task_metadata.target_column_name

for repeat_id, folds in container.experiment_metadata.splits.items():
    for fold_id, (train_idx, test_idx) in folds.items():
        X_train, y_train = df.iloc[train_idx].drop(columns=target), df.iloc[train_idx][target]
        X_test,  y_test  = df.iloc[test_idx].drop(columns=target),  df.iloc[test_idx][target]
        # ... fit, evaluate ...

Full worked example (Random Forest, RMSE per fold, full metadata via container.describe()): examples/benchmark_on_beyond_arena.py.

Split regimes. A collection ships datasets from three regimes — which one a dataset is in shows up directly on task_metadata:

Container formats. container.format_version is 2 for a container built by a dataset.py definition (the TabArena v0.2 working copy and every new dataset) and 1 for the shipped BeyondArena containers, which were built with Data Foundry v1 (DATA_FOUNDRY_V1.md); a format-2 container also states it in container_metadata.json. The collections API and CuratedContainer.load read both. Both read the same through task_metadata.time_on, group_on, group_labels and group_time_on. Format 2 stores the group fields once, in a Grouping (data_foundry.schema.Grouping), and container.grouping returns it: the prediction unit (a row, or one prediction per group with an aggregation such as mean or any), what a model may know about a group when predicting, and the definition of a group with its source. A format-1 container has no such block, so container.grouping raises there. The group column is split and scoring metadata, never a model feature. run_bundle_checks judges each format by its own split protocol (v1: the repeat ladder and 250k test rows; v2: 3 folds and 500k test rows).

Checksums. container.verify() recomputes a container's checksum and compares it with the stored one (with get_dataset(..., verify=True) a collection does it on load). A container built since 2026-10-06 has a version-2 checksum (v2:<hex>): it also covers each categorical column's categories, their order and ordered (saved in categories.json, since dtypes.json only says category), and the unlabeled test set through its own test_dataset_checksum; a new container needs the index 0..n-1 and string column names. A checksum without the prefix is version 1 (every shipped BeyondArena container) and keeps verifying as it always did.

Side-by-side regime printout (one IID, two grouped variants — per_group vs per_sample — and one temporal): examples/data_foundry_data_regimes.py.

A collection pins its datasets by (unique_name, uuid) and resolves them against a source (Hugging Face, or a local warehouse); list_collections() and get_collection(name) find the registered ones. BEYOND_ARENA is the first, and the TabArena v0.2 collection will be registered the same way. BEYOND_ARENA.prefetch(...) batches every container into a single Hugging Face snapshot_download call (one network round-trip for the whole collection). On a warm cache it skips importing huggingface_hub entirely.

from data_foundry.collections import BEYOND_ARENA

paths = BEYOND_ARENA.prefetch()          # warms the cache once
for container in BEYOND_ARENA.iter_containers():  # now hits disk only
    print(container.dataset_metadata.unique_name, container.dataset.shape)

Cache management:

BEYOND_ARENA.clear_cache()                 # nuke this collection's subdir
BEYOND_ARENA.get_dataset(name, force_download=True)  # re-fetch a single container

Full worked example with tqdm progress + checksum verification: examples/download_all_beyond_arena_datasets.py.

A dataset is one class in a dataset.py: flat attributes for the metadata and the task, and hooks for reading and cleaning. The base class adds the dtype casts, the row order, the split protocol, the checks and the export. A complete small definition (blood_transfusion), shortened:

import pandas as pd
from data_foundry.v2 import AbstractCuratedDataset


class BloodTransfusion(AbstractCuratedDataset):
    unique_name = "blood_transfusion"
    year = "2008"
    domain = "medical & healthcare"
    source = "UCI"
    source_url = "https://doi.org/10.24432/C5GS39"
    license = "CC BY 4.0"
    download_description = """
        mkdir -p local-data-warehouse/blood_transfusion/ && wget -P local-data-warehouse/blood_transfusion/ <UCI zip> && unzip ...
    """
    bibtex = """
        @article{yeh2009knowledge, title={Knowledge discovery on RFM model using Bernoulli sequence}, ...}
    """
    curation_comments = """
        - We renamed the target and mapped binary values to "Yes"/"No".
        - Anomaly: the data has a lot of duplicates (29%) and several duplicates with different target values.
    """
    target = "DonatedBloodInMarch2007"
    problem_type = "binary_classification"   # metric roc_auc and a stratified IID split by default

    def _load_raw(self, raw_dir):
        return pd.read_csv(raw_dir / "transfusion.data")

    def _clean(self, raw):
        df = raw.set_axis([..., "DonatedBloodInMarch2007"], axis=1)
        df["DonatedBloodInMarch2007"] = df["DonatedBloodInMarch2007"].map({1: "Yes", 0: "No"})
        return df

A grouped task adds grouping = Grouping(on=..., labels=..., prediction_unit=..., definition=...), a temporal one temporal = Temporal(on=..., splits=TemporalSplits(...)). Then:

.venv/bin/python -m data_foundry.curation.cli dataset new <unique_name> --root datasets/_dev/tabarena-v0pt2
.venv/bin/python -m data_foundry.curation.cli dataset check datasets/_dev/tabarena-v0pt2/<unique_name>   # no save; writes README.md
.venv/bin/python -m data_foundry.curation.cli dataset build datasets/_dev/tabarena-v0pt2/<unique_name>   # saves the container (curators)

The contributor flow (the steps, the probes, the Claude Code skills, the practices) is in CONTRIBUTING_DATASETS.md.

Before a dataset becomes a definition, it lives in the curation log: one markdown record per candidate dataset under curation/records/. Each <unique_name>.md has YAML front-matter (the structured / dropdown fields) plus a free-text body (## Comments, ## Reference). Add or triage a dataset by creating/editing its file — by hand, with an agent, or through the dashboard.

pip install "data-foundry[curation]"   # or the editable dev install
data-foundry-curation serve            # → http://127.0.0.1:8765

The local, Sheets-like dashboard edits those records in place (filter by status, pin a working row, add dropdown options, …) and ships a built-in Guidelines tab describing the selection criteria and processing conventions. Other CLI subcommands:

data-foundry-curation validate                 # check records against the dropdown vocab
data-foundry-curation sync-notebooks           # refresh each record's notebook_path + v2_path (--check to dry-run)
data-foundry-curation export --format xlsx out.xlsx   # flat snapshot (csv|parquet|xlsx|gsheet)
data-foundry-curation build-site site/          # read-only static copy (e.g. GitHub Pages)
data-foundry-curation dataset check <folder>    # run a v2 dataset.py + bundle checks, write README.md (no save)

Browse it online. A read-only copy of the backlog is published to GitHub Pages — tabarena.github.io/data-foundry — and regenerated automatically from curation/records/ on every push to main (no install, no network round-trip to Hugging Face; search, sort, filter, and pin all run in the browser). Note this makes every record's comments, reviewer names, and decision notes public.

Working with Claude Code? The repo ships five curation workflows (in .claude/skills/):

Loading, browsing and benchmarking shipped datasets needs no command; CLAUDE.md and examples/ cover it.

pip install data-foundry            # load and use curated datasets
pip install "data-foundry[build]"   # also build datasets from their definitions

git clone https://github.com/TabArena/data-foundry.git
cd data-foundry
uv pip install -e .

git clone https://github.com/TabArena/data-foundry.git
cd data-foundry
uv pip install -e ".[dev]"
pytest                                 # run the test suite
ruff check . && ruff format --check .  # lint + format

Loading and using a container needs only the core install. The build extra adds what building a dataset from its definition needs: scikit-learn for the splits, scipy for the checks, and the readers the definitions use (liac-arff, openml, polars, the Excel engines, matplotlib). tests adds build and pytest; dev adds both plus the curation tools (autogluon, kaggle, seaborn, ruff, …). pandas 2.3 and 3 build and load the same containers; the lock stays on pandas 2 because AutoGluon (and so TabArena) requires pandas below 2.4.