Lightweight Distributed ML Experiments Management 🛠️

Coming up with the right hypothesis is hard - testing it should be easy.

ML researchers need to coordinate different types of experiments on separate remote resources. The Machine Learning Experiment (MLE)-Toolbox is designed to facilitate the workflow by providing a simple interface, standardized logging, many common ML experiment types (multi-seed/configurations, grid-searches and hyperparameter optimization pipelines). You can run experiments on your local machine, high-performance compute clusters (Slurm and Sun Grid Engine) as well as on cloud VMs (GCP). The results are archived (locally/GCS bucket) and can easily be retrieved or automatically summarized/reported.

What Does The mle-toolbox Provide? 🧑‍🔧

1. API for launching jobs on cluster/cloud computing platforms (Slurm, GridEngine, GCP).
2. Common machine learning research experiment setups:
   • Launching and collecting multiple random seeds in parallel/batches or async.
   • Hyperparameter searches: Random, Grid, SMBO, PBT, Nevergrad, etc.
   • Pre- and post-processing pipelines for data preparation/result visualization.
3. Automated report generation for hyperparameter search experiments.
4. Storage/retrieval of results and database in Google Cloud Storage Bucket.
5. Resource monitoring with dashboard visualization.

The 4 Step mle-toolbox Cooking Recipe 🍲

1. Follow the instructions below to install the mle-toolbox and set up your credentials/configuration.
2. Learn more about the individual infrastructure subpackages with the dedicated tutorial.
3. Read the docs explaining the pillars of the toolbox & the experiment meta-configuration job .yaml files .
4. Check out the example workflows 📄 to get started.
5. Run your own experiment using the template files/project and mle run.

Installation ⏳

If you want to use the toolbox on your local machine follow the instructions locally. Otherwise do so on your respective cluster resource (Slurm/SGE). A PyPI installation is available via:

pip install mle-toolbox

If you want to get the most recent commit, please install directly from the repository:

pip install git+https://github.com/mle-infrastructure/mle-toolbox.git@main

The Core Toolbox Subcommands 🌱

You are now ready to dive deeper into the specifics of experiment configuration and can start running your first experiments from the cluster (or locally on your machine) with the following commands:

You can find more documentation for each subcommand here.

Examples 📄 & Notebook Walkthroughs 📓

	Description	Colab
📓 Getting Started	Get started with the toolbox.
📓 Subpackages	Get started with the toolbox subpackages.
📓 MLExperiment	Introduction to MLExperiment wrapper.
📓 Evaluation	Evaluation of gridsearch results.
📓 GIF Animations	Walk through a set of animation helpers.
📓 Testing	Perform hypothesis tests on logs.

Acknowledgements & Citing the MLE-Infrastructure ✏️

If you use parts the mle-toolbox in your research, please cite it as follows:

@software{mle_infrastructure2021github,
  author = {Robert Tjarko Lange},
  title = {{MLE-Infrastructure}: A Set of Lightweight Tools for Distributed Machine Learning Experimentation},
  url = {http://github.com/mle-infrastructure},
  year = {2021},
}

Development 👷

You can run the test suite via python -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to create an issue and/or start contributing 🤗.