Skip to main content

Hyperparameter optimization via gradient boosting regression

Project description

SpaceOpt: hyperparameter optimization via gradient boosting regression

Python PyPI version license

SpaceOpt is a hyperparameter optimization algorithm that uses gradient boosting regression to find the most promising candidates for the next trial by predicting their evaluation score.

Installation

$ pip install spaceopt

Usage

  1. Define a discrete hyperparameter search space, for example:
search_space = {
    'a': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],  # list of ordered numbers: ints
    'b': [-3.5, -0.1, 0.0, 2.5, 10.0],    # list of ordered numbers: floats
    'c': [256, 512, 1024, 2048],          # another list of ordered numbers
    'd': ['ABC', 'IJK', 'XYZ'],           # categorical variable
    'e': [True, False],                   # boolean variable
    # ... (add as many as you need)
}
  1. Define your evaluation function:
def evaluation_function(spoint: dict) -> float:
    # your code (e.g. model fit)
    return y  # score (e.g. model accuracy)

spoint = {'a': 4, 'b': 0.0, 'c': 512, 'd': 'XYZ', 'e': False}
y = evaluation_function(spoint)
  1. Use SpaceOpt for a hyperparameter optimization:
from spaceopt import SpaceOpt

spaceopt = SpaceOpt(search_space=search_space,
                    target_name='y',
                    objective='maximize')  # or 'minimize'

for iteration in range(200):
    if iteration < 20:
        spoint = spaceopt.get_random()     # exploration
    else:
        spoint = spaceopt.fit_predict()    # exploitation

    spoint['y'] = evaluation_function(spoint)
    spaceopt.append_evaluated_spoint(spoint)

More examples here.

Advanced

  • get multiple points by setting num_spoints:
spoints = spaceopt.get_random(num_spoints=2)
# or
spoints = spaceopt.fit_predict(num_spoints=5)
  • control exploration vs. exploitation behaviour by adjusting sample_size (default=10000), which is the number of candidates sampled for ranking:
spoint = spaceopt.fit_predict(sample_size=1000)  # decreasing `sample_size` increses exploration
spoint = spaceopt.fit_predict(sample_size=100000)  # increasing `sample_size` increses exploitation
  • add manually selected evaluation points to SpaceOpt:
my_spoint = {'a': 8, 'b': -3.5, 'c': 256, 'd': 'IJK', 'e': False}
my_spoint['y'] = evaluation_function(my_spoint)
spaceopt.append_evaluated_spoint(my_spoint)

License

MIT License (see LICENSE).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spaceopt-0.2.2.tar.gz (7.7 kB view hashes)

Uploaded source

Built Distribution

spaceopt-0.2.2-py3-none-any.whl (7.8 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page