XBBO 0.2.1

A Python toolbox for performing Black-Box Optimization.

XBBO

XBBO is an an effective, modular, reproducible and flexible black-box optimization (BBO) codebase, which aims to provide a common framework and benchmark for the BBO community.

Installation

Python >= 3.7 is required.

For pip

pip install xbbo

For Development

git clone REPO_URL
cd XBBO
# install requirements
pip install -r ./requirements.txt
# set root path
export PYTHONPATH=$PYTHONPATH:/Path/to/XBBO

Quick Start

note:XBBO default minimize black box function.

Bayesian Optimization example

Script path is ./examples/optimize_api_rosenbrock_bo.py

import numpy as np

from xbbo.search_space.fast_example_problem import build_space_hard, rosenbrock_2d_hard
from xbbo.search_algorithm.bo_optimizer import BO
from xbbo.utils.constants import MAXINT

if __name__ == "__main__":
  MAX_CALL = 30
  rng = np.random.RandomState(42)

  # define black box function
  blackbox_func = rosenbrock_2d_hard
  # define search space
  cs = build_space_hard(rng)
  # define black box optimizer
  hpopt = BO(space=cs,
              objective_function=blackbox_func,
              seed=rng.randint(MAXINT),
              suggest_limit=MAX_CALL,
              initial_design='sobol',
              surrogate='gp',
              acq_opt='rs_ls')

  # ---- Use minimize API ----
  hpopt.optimize()
  best_value, best_config = hpopt.trials.get_best()
  print('Find best value:{}'.format(best_value))
  print('Best Config:{}'.format(best_config))

This example shows how to use this .optimize() api to easily and quickly optimize a black box function.

Script path is ./examples/rosenbrock_bo.py

def build_space(rng):
    cs = ConfigurationSpace(seed=rng.randint(MAXINT))
    x0 = UniformFloatHyperparameter("x0", -5, 10, default_value=-3)
    x1 = UniformFloatHyperparameter("x1", -5, 10, default_value=-4)
    cs.add_hyperparameters([x0, x1])
    return cs

rng = np.random.RandomState(42)
# define black box function
blackbox_func = rosenbrock_2d
# define search space
cs = build_space(rng)
# define black box optimizer
hpopt = BO(config_spaces=cs, seed=rng.randint(MAXINT), suggest_limit=MAX_CALL)
# Example call of the black-box function
def_value = blackbox_func(cs.get_default_configuration())
print("Default Value: %.2f" % def_value)
# ---- Begin BO-loop ----
for i in range(MAX_CALL):
    # suggest
    trial_list = hpopt.suggest()
    # evaluate 
    value = blackbox_func(trial_list[0].config_dict)
    # observe
    trial_list[0].add_observe_value(observe_value=value)
    hpopt.observe(trial_list=trial_list)
  
    print(value)  

This example shows how to use .ask()、.tell() api to quickly optimize a black box function.

All examples can be found in examples/ folder.

Supported Algorithms

• Transfer

  • TST-R
  • TAF
  • TAF(RGPE)
  • RMoGP
  • RGPE(mean)

• Optimizer

  • BORE optimizer
  • Anneal
  • DE
  • CMA
  • NSGA
  • Regularized EA
  • PBT
  • TuRBO

• multi-fidelity

  • HyperBand
  • BOHB
  • DEHB
  • MFES-BO

Benchmark

Run comparison/xbbo_benchmark.py to benchmark general BBO optimizer.

Method	Minimum	Best minimum	Mean f_calls to min	Std f_calls to min	Fastest f_calls to min
XBBO(rs)	0.684+/-0.248	0.399	110.4	60.511	17
XBBO(bo-gp)	0.398+/-0.000	0.398	138.5	33.685	90
XBBO(tpe)	0.519+/-0.119	0.398	191.4	12.035	162
XBBO(anneal)	0.404+/-0.005	0.399	164.5	29.032	92
XBBO(cma-es)	0.398+/-0.000	0.398	191.3	8.391	174
XBBO(rea)	0.425+/-0.026	0.399	115.8	47.743	56
XBBO(de)	0.465+/-0.065	0.399	163.5	27.969	99
XBBO(turbo-1)	0.398+/-0.000	0.398	110.3	46.596	46
XBBO(turbo-2)	0.398+/-0.000	0.398	130.7	48.57	68
XBBO(bore)	0.408+/-0.006	0.401	117.4	58.114	38
XBBO(cem)	1.875+/-2.090	0.398	144.8	60.834	36

Compare other bbo library

Here you can comparison with commonly used and well-known Hyperparameter Optimization (HPO) packages:

SMAC3

hyperopt

scikit-optimize

TuRBO

Bayesian Optimization

DEHB、HpBandSter

OpenBox

Hypermapper

Algorithms notes

• BORE
• CEM
• CMA-ES
• DE
• NSGA-II
• PBT
• REA
• TuRBO
• TPE
• TST-R
• RGPE、RMoGP、TAF
• TAF
• HyperBand
• BOHB

review

TODO

• parallel
• multi-fidelity