researchlab 0.0.2

Lightweight dataset/model benchmarking framework for ML experiments

researchlab

researchlab is a lightweight benchmarking framework that runs every (dataset, model) pair in your experiment grid, skips pairs that are already done, and stores results as plain CSV files — locally or on S3.

It ships two ready-made runners:

Runner	What it evaluates
SklearnClassificationRunner	Any sklearn-compatible classifier with K-fold CV
SkproRegressionRunner	Any skpro probabilistic regressor with proper scoring rules

You can also plug in any library or custom evaluation logic by subclassing LocalRunner or BaseRunner.

Install

Core package (sklearn benchmarking, local storage):

pip install researchlab

With skpro probabilistic regression support:

pip install "researchlab[skpro]"

With AWS S3 storage and Batch/ECS submission:

pip install "researchlab[aws]"

Everything:

pip install "researchlab[all]"

Quick start — sklearn classification

# experiment.py
from pathlib import Path
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import get_scorer
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from researchlab import Dataset, DatasetPayload, Model, SklearnClassificationRunner


class BreastCancer(Dataset):
    name = "breast_cancer"

    def _load(self):
        b = load_breast_cancer(as_frame=True)
        return DatasetPayload(data=(b.data, b.target.to_frame(name="target")))


class LogReg(Model):
    name = "logistic_regression"

    def _instantiate(self, metadata):
        return make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))


runner = SklearnClassificationRunner(
    dataset_collection={"breast_cancer": BreastCancer()},
    model_collection={"logistic_regression": LogReg()},
    artifact_repository=Path("results/"),
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    metrics={"accuracy": get_scorer("accuracy"), "roc_auc": get_scorer("roc_auc")},
)

if __name__ == "__main__":
    runner.cli()

python experiment.py run                    # run all pairs, skip done
python experiment.py run --count            # show done / remaining
python experiment.py run -d breast_cancer   # one dataset only
python experiment.py concatenate -o out.csv # merge all results

Quick start — skpro probabilistic regression

from pathlib import Path
from sklearn.datasets import load_diabetes
from sklearn.model_selection import KFold
from skpro.regression.residual import ResidualDouble
from sklearn.linear_model import LinearRegression

from researchlab import Dataset, DatasetPayload, Model, SkproRegressionRunner


class Diabetes(Dataset):
    name = "diabetes"

    def _load(self):
        b = load_diabetes(as_frame=True)
        return DatasetPayload(data=(b.data, b.target.to_frame(name="target")))


class LinearResidual(Model):
    name = "linear_residual"

    def _instantiate(self, metadata):
        return ResidualDouble(LinearRegression())


runner = SkproRegressionRunner(
    dataset_collection={"diabetes": Diabetes()},
    model_collection={"linear_residual": LinearResidual()},
    artifact_repository=Path("results/"),
    cv=KFold(n_splits=5, shuffle=True, random_state=42),
    metrics=["CRPS", "LogLoss"],
)

if __name__ == "__main__":
    runner.cli()

Scaling out to AWS

Add the AWSRunner mixin to any runner to get aws batch and aws ecs subcommands — no separate submission script needed:

from researchlab.aws import AWSRunner, S3ArtifactRepository

class MyAwsRunner(AWSRunner, SklearnClassificationRunner):
    pass

runner = MyAwsRunner(
    ...
    artifact_repository=S3ArtifactRepository("s3://my-bucket/my-experiment"),
    container_command=["python", "/app/experiment.py"],
)

# submit every pair as a Batch job from your laptop
python experiment.py aws batch \
    --job-queue my-queue \
    --job-definition my-jobdef

# collect results after jobs finish
python experiment.py concatenate -o results.csv

Examples

Example	What it shows
examples/researchlab_classification/	Local sklearn benchmark
examples/researchlab_regression/	Local skpro benchmark
examples/researchlab_aws_classification/	sklearn + S3 + Batch/ECS
examples/researchlab_aws_regression/	skpro + S3 + Batch/ECS

Documentation

Full documentation lives in docs/ and is built with Quarto.

quarto render docs/

Contributing

1. Fork the repository
2. Install dev dependencies: poetry install --extras all --with dev
3. Run tests: poetry run pytest
4. Open a pull request against main — CI runs automatically

License

BSD 3-Clause

metrics = [RMSE()]

cv = KFold(n_splits=3, shuffle=True, random_state=42)

RUNNER = ExperimentRunner() SAVE_BEST_MODEL = True SAVE_BEST_MODEL_PICKLE = True

To store artifacts outside the experiment directory, set `ARTIFACT_ROOT` in
`experiments_config.py`.

- Local path: `ARTIFACT_ROOT = "/absolute/path/to/results"`
- Relative local path: `ARTIFACT_ROOT = "shared-results"`
- S3 path: `ARTIFACT_ROOT = "s3://my-bucket/experiments/exp1"`

If `ARTIFACT_ROOT` is omitted, storage defaults to the local directory
`<experiment_dir>/results/`.

If you need custom orchestration, define `RUNNER` or `runner` in
`experiments_config.py` with an instance of a `BaseRunner` subclass. The
built-in `ExperimentRunner` preserves the current behavior and can be
subclassed by overriding hooks such as config validation, dataset preparation,
task inference, evaluation, summary aggregation, or best-model selection.

## CLI

Run an experiment:

```bash
poetry run probabilistic_experiment path/to/experiment

Validate only:

poetry run probabilistic_experiment path/to/experiment --dry-run

Run specific models:

poetry run probabilistic_experiment path/to/experiment --model model_a --model model_b

Run specific datasets:

poetry run probabilistic_experiment path/to/experiment --dataset dataset_a --dataset dataset_b

Re-run datasets with existing outputs:

poetry run probabilistic_experiment path/to/experiment --include-evaluated

Concatenate all per-model fold outputs:

poetry run probabilistic_experiment concat path/to/experiment

If --output is omitted, the concatenated CSV is written to concatenated_results.csv under the active artifact root.

Erase a model's stored artefacts and refresh summaries:

poetry run probabilistic_experiment path/to/experiment --erase-model model_a

Build and submit the AWS Batch example:

poetry run probabilistic_experiment aws build-image
poetry run probabilistic_experiment aws register-job-definition --image-uri ... --execution-role-arn ... --job-role-arn ...
poetry run probabilistic_experiment aws submit examples/aws_batch_local_classification --job-queue ... --job-definition ... --artifact-root s3://...

Output Layout

The runner writes results to path/to/experiment/results/<dataset>/:

• <model>.csv: standardized fold-level results
• <model>_data.pkl: pickled fold data and predictions
• metrics.csv: dataset-level aggregated summary for all remaining models
• best_model.json: metadata for the best available model
• best_model.pkl: optional fitted best model pickle

If ARTIFACT_ROOT is configured, the same layout is written under that local or S3 root instead of path/to/experiment/results/.

The runner uses a storage abstraction internally, so reads, writes, deletion, and listing all go through the same backend interface. Local filesystem storage is the default implementation; S3 is available by setting ARTIFACT_ROOT to an s3://... URI.

The fold-level CSVs always include:

• dataset
• model
• task
• fold
• status
• fit_time
• pred_time
• one or more test_<metric> columns

More

• Example config: example/exp1/experiments_config.py
• Local runnable example: examples/local_classification/README.md
• Detailed usage guide: docs/USAGE.md
• Artifact storage guide: docs/ARTIFACTS.md
• AWS Batch parallel tutorial: docs/AWS_BATCH_TUTORIAL.md
• Runner architecture: experiment/init.py
• Storage API module: artifacts.py