Simple HPO Benchmark Datasets

This repository provides a set of simple single-objective HPO benchmark datasets:

• HPOBench (Apache-2.0 License)
• HPOLib (BSD-3-Clause License)
• NAS-Bench-201 (MIT License)

Installation & Requirements

The requirements of this repository are:

• Python 3.8 or later
• NumPy

You can simply install the package via:

$ pip install simple-hpo-bench

Examples

Examples are available at examples/.

For example, HPOBench can be optimized by Optuna as follows:

from __future__ import annotations

from hpo_benchmarks import HPOBench

import optuna


# Instatiate the benchmark function.
bench = HPOBench(dataset_name="australian")


def objective(trial: optuna.Trial) -> list[float]:
    params = {}
    for param_name, choices in bench.search_space.items():
        params[param_name] = choices[trial.suggest_int(f"{param_name}_index", low=0, high=len(choices) - 1)]

    results = bench(params)
    return [results[name] for name in bench.metric_names]


study = optuna.create_study(direction=[bench.directions[name] for name in bench.metric_names])
study.optimize(objective, n_trials=30)

For benchmark functions, the argument for these classes is only dim, which determines the dimension of the function.

For tabular benchmarks (HPOLib, HPOBench, and NASBench201), the arguments are:

• dataset_name: one of the dataset names of the benchmark dataset of interest, and
• seed: the random seed for the benchmark dataset.

The available dataset names for each benchmark dataset are as follows:

Search Space

In this section, bench is an instantiated benchmark.

The search space of the benchmark functions are [bench.param_range[0], bench.param_range[1]]^bench.dim.

The search space of HPOLib, HPOBench, and NASBench201 is defined in bench.search_space. bench.search_space is a dict that takes param_name as a key and the corresponding possible choices as a value. bench.param_types defines the types of each parameter. int is for an integer parameter, str is for a categorical parameter, and float is for a non-integer numerical parameter.