researchlab 0.0.3

Lightweight dataset/model benchmarking framework for ML experiments

researchlab

Research experiments are made of many layers and some pre-defined steps. Some common pre-defined artifacts:

• Models
• Datasets
• Evaluation Policy: how to cross-validation, how to evaluate, etc, when to instantiate and fit models
• Metrics

And some output artifacts:

• Checkpoints
• Logs
• Plots
• Metrics

This package provides a framework to define and run such experiments and to manage artifacts in a modular fashion. The structure allows easy use with different backends and helps with reproducibility and organization of experiments.

Components

For every component, the init arguments are mapped to public attributes with the exact same name and values provided at initialization. This allows for easy access to the configuration of each component and promotes transparency in the experiment setup. Pydantic is used to enforce type checking and validation of attributes. A get_params method, similar to scikit-learn, skbase and sktime, is provided with recursive support for nested components. This method returns a dictionary of all parameters and their values, allowing for easy inspection and debugging of the experiment configuration.

Evaluation Policy contract

Evaluation policy receives has a method:

• evaluate(model, dataset, metrics): returns a dictionary of metric names and their values, given a model, a dataset and a list of metrics. The metrics are returned in a list for each key since evaluation policies usually involve cross-validation and thus return multiple values for each metric.

For a scikit-learn experiment, the cv is passed in __init__ and the evaluate method performs cross-validation using the provided cv and returns the metrics for each fold.

Evaluate can also return a key "_extra_info" with extra information.

Models contract

A Model is class that implements the following method:

• instantiate: returns an instance of the model, ready to be fitted.

Each model has a name attribute that is used to identify the model in the results and artifacts. The name should be unique across different models used in the same experiment to avoid confusion in the results and artifacts. The name can be used in the storage to organize artifacts by model, making it easier to track and compare results across different models.

Dataset contract

A Dataset is a class that implements the following method:

• load: returns the data.

Metrics contract

A Metric is a class that implements the following method:

• evaluate(fitted_model, **kwargs): returns the value of the metric given a fitted model and any additional arguments needed for evaluation (e.g., test data, true labels, etc.). Returns a float or a dictionary of metric values.

Storage contract

A Storage is a class that implements the following methods:

• save(key, value): saves a value exactly under the given key and must not create additional sidecar artifacts implicitly.
• load(key): loads the value stored exactly at that key.
• exists(key): returns True if that exact key exists in storage.
• erase(key): removes only the value stored under that exact key.

Keys may contain slashes (e.g., "dataset/model") which backends may map to subdirectories. Artifact formats are part of the key, e.g. "dataset/model.pkl" and "dataset/model.csv".

Runner

Runner consolidates the evaluation policy, models, datasets, metrics, and storage to run the experiment. It provides a CLI via runner.cli() so that, when instantiated in a script, the experiment can be run from the command line with run, list, and erase commands accepting --dataset and --model filters.

Results from the evaluation policy are saved under two explicit keys per run: "{dataset.name}/{model.name}.csv" for the tabular metrics view and "{dataset.name}/{model.name}.pkl" for the full Python result object. The storage can be local (e.g., LocalStorage) or any custom backend implementing the Storage contract.