chronobench 0.1.2

A flexible benchmarking framework for ML models.

Chronobench

A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.

Installation

Install Chronobench as follows:

pip install chronobench

Quick Start

Keep your components in plain modules (here loaders.py), then write a script.py that registers them, defines a runner, and calls chronobench.main():

import chronobench as cb
from loaders import IrisLoader, BreastCancerLoader

cb.data_loaders(iris=IrisLoader, breast_cancer=BreastCancerLoader)

@cb.runner
def runner(data_loader, model, metrics) -> dict:
    X_train, X_test, y_train, y_test = data_loader.load()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    return {m.func.__name__: m(y_test, y_pred) for m in metrics}

if __name__ == "__main__":
    cb.main()

Experiment TOML entries pick an implementation by registered name or by dotted target import path:

[[data]]
name = "iris"                              # a registered name
test_size = 0.2

[[models]]
target = "sklearn.svm.SVC"                 # an import path - no registration
kernel = "linear"

[[metrics]]
target = "sklearn.metrics.accuracy_score"  # an import path — no registration

Then run it from the command line:

python script.py my_experiment

See examples/simple/ for a complete working example with scikit-learn.

Concepts

Registering components

Chronobench keeps three registries — cb.data_loaders, cb.models, and cb.metrics — plus one runner. A factory is any callable that produces a component: a class, or a function defined in your script. Call a registry to register factories — either with keyword pairs or as a decorator:

cb.data_loaders(<short-name>=<object>)   # keyword pairs

@cb.data_loaders                          # bare decorator — name taken from the function
def iris(...): ...

@cb.data_loaders("all")                   # decorator with an explicit name
def all_datasets(...): ...

Registration is optional: anything importable can be referenced by its target path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.

Because data_loader and model are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as runner knows how to use it. The runner returns a dict whose keys become CSV columns.

name vs target

Each TOML entry selects its implementation with exactly one of:

• name — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
• target — a dotted import path (e.g. "sklearn.svm.SVC"). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.

All other keys in the entry are forwarded to the factory as **kwargs.

Factories return lists or single instances

Each factory may return a single instance or a list of instances. A single instance produces one job variant, a list produces one variant per element, each crossed with every other-side variant. This allows you to easily run multiple datasets and/or models in the same experiment without writing any loops or separate config files.

A factory may return a list of instances. A single [[data]] (or [[models]]) entry then fans out into one variant per element, each crossed with every model (resp. data) variant — i.e. one CSV row per (item × other-side-config). Each row is labeled by the instance's name attribute when present, otherwise by the entry label plus an index.

@cb.data_loaders("all")   # one [[data]] entry -> two dataset rows
def all_datasets(test_size=0.2, random_state=None):
    return [
        IrisLoader(test_size=test_size, random_state=random_state),
        BreastCancerLoader(test_size=test_size, random_state=random_state),
    ]

To instead keep a single job that internally manages several splits (e.g. a tuning + evaluation set in one folder), just return one object whose load() exposes all splits and let runner use them.

Metrics: scoring functions without wrappers

A metric is a callable the runner invokes with data (e.g. metric(y_true, y_pred)). You can use a class whose instance is that callable, but a bare scoring function works directly — by target (no registration) or by a registered name:

[[metrics]]
target = "sklearn.metrics.fbeta_score"
beta = 1.0
average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner

When the config kwargs don't satisfy every required parameter (here fbeta_score still needs y_true/y_pred), Chronobench binds the given arguments with functools.partial and lets the runner supply the rest, instead of calling the function immediately. The result is a bound function, so name the CSV column from it in your runner (e.g., metric.func.__name__). Note that a bare function uses its own defaults (e.g., fbeta_score defaults to average="binary", which fails on multiclass data), so set what you need in the TOML.

To register such a function under a short name instead, the optional convenience is:

from sklearn.metrics import accuracy_score, fbeta_score
cb.metrics(accuracy=accuracy_score, fbeta=fbeta_score)

Context and environment injection

Factories are called with their TOML **kwargs. Chronobench also passes environment and/or context only when the factory declares them as explicit parameters, so plain and library classes work untouched while advanced factories can opt in:

def make_model(context, n_layers=2):     # receives the cross-initializer context
    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)

environment is the parsed environment.toml. context is a Context dataclass built progressively across stages:

Stage	Populated context fields
data loaders	none
models	data_loader, data_loader_name, data_loader_kwargs
metrics	above + model, model_name, model_kwargs

name, target, context, and environment are reserved keys and must not be used as TOML parameter names.

Configuration files

environment.toml — global settings, required keys:

path-to-results     = "./results"
path-to-experiments = "./experiments"
n-jobs              = 4

experiments/<name>.toml — one file per experiment, with [[data]], [[models]], and [[metrics]] arrays-of-tables. [[data]] and [[models]] entries produce job variants (their Cartesian product is the final job list); [[metrics]] entries are collected into a single flat list used for every job.

Sweep parameters

Add sweep.<key> = [v1, v2, ...] to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on [[metrics]] entries only adds more metric instances per job — it does not create additional jobs.

[[data]]
name = "iris"
sweep.random_state = [0, 1, 2]   # 3 variants

[[models]]  # 2 × 2 = 4 model variants
target = "sklearn.ensemble.RandomForestClassifier"
sweep.criterion = ["gini", "entropy"]  # 2 variants
sweep.max_depth  = [5, 10]             # 2 variants

[[metrics]]
target = "sklearn.metrics.fbeta_score"
average = "macro"
sweep.beta = [0.5, 1.0, 2.0]  # 3 variants

This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in {0.5, 1.0, 2.0}$).

Results

Results are written to:

{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv

Columns are the union of all keys returned by runner(), plus Dataset and Model metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with Dataset. / Model. to avoid collision.

Run python script.py --help for the full list of options.

Roadmap

This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the issue page for an overview of the tasks we plan to add in the near future.