Chronobench

A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.

Installation

Install Chronobench as follows:

pip install chronobench

Quick Start

Below example shows a minimal working experiment with scikit-learn. A data loader for the Iris dataset is registered, a runner is defined, and chronobench is started:

import chronobench as cb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split


@cb.data_loaders("iris")
class IrisLoader:
    def __init__(self, test_size: float = 0.2, random_state: int | None = None):
        self.test_size = test_size
        self.seed = random_state

    def load(self):
        X, y = load_iris(return_X_y=True)
        return train_test_split(X, y, test_size=self.test_size, random_state=self.seed)


@cb.runner
def runner(data_loader, model, metrics) -> dict:
    X_train, X_test, y_train, y_test = data_loader.load()
    y_pred = model.fit(X_train, y_train).predict(X_test)
    results = {name: metric(y_test, y_pred) for name, metric in metrics.items()}
    return results


if __name__ == "__main__":
    cb.main()

Experiment TOML entries pick an implementation by registered name or by dotted target import path:

[[data]]
name = "iris"                              # a registered name
test_size = 0.2

[[models]]
target = "sklearn.svm.SVC"                 # an import path - no registration
kernel = "linear"

[[metrics]]
target = "sklearn.metrics.accuracy_score"  # an import path — no registration

Then run it from the command line:

python script.py my_experiment

See examples/ for complete working examples.

Concepts

Registering components

Chronobench keeps three registries — cb.data_loaders, cb.models, and cb.metrics — plus one runner. A factory is any callable that produces a component: a class, or a function defined in your script. Call a registry to register factories — either with keyword pairs or as a decorator:

cb.data_loaders(<short-name>=<object>)   # keyword pairs

@cb.data_loaders                          # bare decorator — name taken from the function
def iris(...): ...

@cb.data_loaders("all")                   # decorator with an explicit name
def all_datasets(...): ...

Registration is optional: anything importable can be referenced by its target path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.

Because data_loader and model are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as runner knows how to use it. The runner returns a dict whose keys become CSV columns.

name vs target

Each TOML entry selects its implementation with exactly one of:

• name — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
• target — a dotted import path (e.g. "sklearn.svm.SVC"). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.

All other keys in the entry are forwarded to the factory as **kwargs.

Fan-out: many variants from one entry

A factory may return a single instance (one job variant) or several instances, which fan out into one variant per element — each crossed with every other-side variant, i.e. one CSV row per (item × other-side-config). This runs multiple datasets and/or models in one experiment without loops or separate config files. There are three ways to return several, differing only in how each variant is labeled in the final resutls (its Dataset / Model column):

A list — labeled by the entry label plus an index (all[0], all[1]):

@cb.data_loaders("all")   # one [[data]] entry -> two dataset rows
def all_datasets(test_size=0.2, random_state=None):
    return [
        IrisLoader(test_size=test_size, random_state=random_state),
        BreastCancerLoader(test_size=test_size, random_state=random_state),
    ]

A dict — keys become the labels:

@cb.data_loaders("named")
def named_datasets(test_size=0.2):
    return {
        "iris": IrisLoader(test_size=test_size),
        "breast_cancer": BreastCancerLoader(test_size=test_size),
    }

cb.variant(instance, name=..., **params) — for full control: name sets the label and any extra keyword arguments are recorded as CSV columns. Because the distinguishing value is a recorded parameter rather than part of the label, several variants can share one label yet stay distinct. The following example illustrates how you can create a model with 10 different random seeds programmatically, without having to define this in the configuration file. In the final results file, the "Model" column will be the same for all 10 variants, but each will have a different seed column.

@cb.models("my-seeded-model")
def my_model(**kwargs): 
    n_seeds = 10
    return [
        cb.variant(MyModel(seed=i, **kwargs), name="my_model", seed=i)
        for i in range(n_seeds)
    ]

Metrics: scoring functions without wrappers

A metric is a callable the runner invokes with data (e.g. metric(y_true, y_pred)). You can use a class whose instance is that callable, but a bare scoring function works directly — by target (no registration) or by a registered name:

[[metrics]]
target = "sklearn.metrics.fbeta_score"
beta = 1.0
average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner

When the config kwargs don't satisfy every required parameter (here fbeta_score still needs y_true/y_pred), Chronobench binds the given arguments with functools.partial and lets the runner supply the rest, instead of calling the function immediately. Note that a bare function uses its own defaults (e.g., fbeta_score defaults to average="binary", which fails on multiclass data), so set what you need in the TOML.

The runner receives the metrics as a dict keyed by name — the metric's registered name, the target's last component (e.g. fbeta_score), or an explicit label from cb.variant(metric, name=...). Each name becomes the metric's result column, so it must be unique. Because a metric is a column rather than a row, cb.variant on a metric accepts only a name: passing extra parameters (as you would for a data loader or model row) raises an error.

To register such a function under a short name instead, the optional convenience is:

from sklearn.metrics import accuracy_score, fbeta_score
cb.metrics(accuracy=accuracy_score, fbeta=fbeta_score)

Context and environment injection

Factories are called with their TOML **kwargs. Chronobench also passes environment and/or context only when the factory declares them as explicit parameters, so plain and library classes work untouched while advanced factories can opt in:

def make_model(context, n_layers=2):     # receives the cross-initializer context
    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)

environment is the parsed environment.toml. context is a Context dataclass built progressively across stages:

name, target, context, and environment are reserved keys and must not be used as TOML parameter names.

Configuration files

environment.toml — global settings, required keys:

path-to-results     = "./results"
path-to-experiments = "./experiments"
n-jobs              = 4

experiments/<name>.toml — one file per experiment, with required [[data]] and [[models]] arrays-of-tables and an optional [[metrics]] array. [[data]] and [[models]] entries produce job variants (their Cartesian product is the final job list); [[metrics]] entries are collected into a single flat list used for every job. Omit [[metrics]] entirely for experiments that compute no metric — e.g. scaling studies where the runner measures and returns runtime itself (the runner then receives an empty metrics list).

Sweep parameters

Add sweep.<key> = [v1, v2, ...] to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on [[metrics]] entries only adds more metric instances per job — it does not create additional jobs.

[[data]]
name = "iris"
sweep.random_state = [0, 1, 2]   # 3 variants

[[models]]  # 2 × 2 = 4 model variants
target = "sklearn.ensemble.RandomForestClassifier"
sweep.criterion = ["gini", "entropy"]  # 2 variants
sweep.max_depth  = [5, 10]             # 2 variants

[[metrics]]
target = "sklearn.metrics.fbeta_score"
average = "macro"
sweep.beta = [0.5, 1.0, 2.0]  # 3 variants

This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in {0.5, 1.0, 2.0}$).

Results

Results are written to:

{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv

Columns are the union of all keys returned by runner(), plus Dataset and Model metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with Dataset. / Model. to avoid collision.

Run python script.py --help for the full list of options.

Roadmap

This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the issue page for an overview of the tasks we plan to add in the near future.