Basht - A Benchmarking Approach for Sustainabile Hyperparameter Tuning

Tuning This repository supplies a job to benchmark hyperparameter tuning.

Start up

Prerequisites

• Ubuntu >= 16.04 (not tested on MAC OS)
• Python >= 3.7.0
• PyTorch

Note: If you run your benchmark on GPU make sure to install Cuda and install the correct PyTorch Version, which fits your Cuda version.

Install

1. Clone the repository with git clone <url>
2. Create a Python environment with python -m venv .venv.
3. Activate your environment with source .venv/bin/activate
4. Upgrade pip with pip install pip --upgrade
5. If not already installed install PyTorch and Torchvision
6. To install the benchmark and use it locally , clone the repository and switch to the root_folder of the repository and type in pip install -e .. Otherwise you can also install the package via PyPi with pip install ml-benchmark

RayTune Example

For running the raytune example experiment add the following after the process above:

1. pip install ray[tune]
2. pip install pandas

Optuna Minikube Example

For minikube, install the requirements.txt in experiments/optuna_minikube if present. Also make sure you installed minikube. Before starting the minikube example you need to start your minikube vm with minikube start. Then execute experiments/optuna_minikube/optuna_minikube_benchmark.py.

Class Explanation

Benchmark Methodolegy

Each implementation uses a common experiment-docker-container that represents the full lifecycle of a benchmarking experiment, see the lifecycle figure.

Lifecycle

The Lifecycle consists of 7 steps, that we describe in detail in the following: .

The docker container stub is located here.

Collaboration

Make sure to fork the repository. For informations on how to fork read here. When you start working on your code, in your forked repositry, make sure to create a new branch with collaborator_name/feature_name. When performing your work assure, that you keep your main branch of your forked repository up to date with the main brain of this repository as it is a direct copy. How to sync your main branch with the one of this repository can be read here. Once your work is done submit a pull request to the original repository (this one). A guide can be found here. For the Pull Request regular rules apply. Provide a short description of the feature you are adding and make sure your code is in a good shape (decent amount of documentation, no extremely long script files). If you use Python it is recommended to use Flake8.

System Components

Adresses all components that perform necessary task in an HPO. E.g. Scheduling, I/O Operations etc. A component is not necessarily one Object within the framework, but it can be a collection of multiple objects performing similar elemental tasks. A fair compairison between HPO Frameworks would map all HPO Frameworks into these components and measure their performed tasks against each other.