hyperactive 2.3.1

A hyperparameter optimization toolbox for convenient and fast prototyping

---

A hyperparameter optimization and meta-learning toolbox for convenient and fast prototyping of machine-learning models.

Master status:
Dev status:
Code quality:
Latest versions:

Hyperactive is primarly a hyperparameter optimization toolkit, that aims to simplify the model-selection and -tuning process. You can use any machine- or deep-learning package and it is not necessary to learn new syntax. Hyperactive offers high versatility in model optimization because of two characteristics:

• You can define any kind of model in the objective function. It just has to return a score/metric that gets maximized.
• The search space accepts not just int, float or str as data types but even functions, classes or any python objects.

For more information, visualization and details about the API check out the
website

---

Main features • Installation • Roadmap • License

---

Main features

• Thoroughly tested code base
• Compatible with any python machine-learning framework
• Optimize:
  • Anything from simple models
    to complex machine-learning-pipelines
  • Multi-level ensembles
  • Deep neural network architecture
  • Other optimization techniques (meta-optimization)
  • Or any function you can specify with this API
• Utilize state of the art optimization techniques like:
  • Simulated annealing
  • Evolution strategy
  • Bayesian optimization
• High performance: Optimizer time is neglectable for most models
• Choose from a variety of different optimization extensions to improve the optimization

Optimization Techniques	Tested and Supported Packages	Optimization Extensions
Local Search: Hill Climbing Stochastic Hill Climbing Tabu Search Simulated Annealing Global Search: Random Search Random Restart Hill Climbing Random Annealing [*] Population Methods: Parallel Tempering Particle Swarm Optimizer Evolution Strategy Sequential Methods: Bayesian Optimization Tree of Parzen Estimators Decision Tree Optimizer [dto]	Machine Learning: Scikit-learn XGBoost LightGBM CatBoost RGF Mlxtend Deep Learning: Tensorflow Keras Pytorch Distribution: Multiprocessing Ray	Position Initialization: Scatter-Initialization [*] Warm-start Resource Allocation: Memory Short term Long term

Installation

The most recent version of Hyperactive is available on PyPi:

pip install hyperactive

Minimal example

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_breast_cancer
from hyperactive import Hyperactive

data = load_breast_cancer()
X, y = data.data, data.target

'''define the model in a function'''
def model(para, X, y):
    '''optimize one or multiple hyperparameters'''
    gbc = GradientBoostingClassifier(n_estimators=para['n_estimators'])
    scores = cross_val_score(gbc, X, y)

    return scores.mean()

'''create the search space and search_config'''
search_config = {
    model: {'n_estimators': range(10, 200, 10)}
}

'''start the optimization run'''
opt = Hyperactive(X, y)
opt.search(search_config, n_iter=20)

Roadmap

v2.0.0:heavy_check_mark:

• Change API
• Ray integration

v2.1.0:heavy_check_mark:

• Save memory of evaluations for later runs (long term memory)
• Warm start sequence based optimizers with long term memory
• Gaussian process regressors from various packages (gpy, sklearn, GPflow, ...) via wrapper

v2.2.0:heavy_check_mark:

• Add basic dataset meta-features to long term memory
• Add helper-functions for memory
  • connect two different model/dataset hashes
  • split two different model/dataset hashes
  • delete memory of model/dataset
  • return best known model for dataset
  • return search space for best model
  • return best parameter for best model

v2.3.0

• Tree-structured Parzen Estimator
• Decision Tree Optimizer
• add "max_sample_size" and "skip_retrain" parameter for sbom to decrease optimization time

v3.0.0

• New API
  • improve distributed computing abilities
  • separate optimizer and n_iter for each job
  • expand usage of objective-function

v3.1.0

• Spiral optimization
• Downhill-Simplex-Method
• upgrade particle swarm optimization
• upgrade evolution strategy
• add warm start for population based optimizers
• Meta-Optimization of local optimizers

Experimental algorithms

The following algorithms are of my own design and, to my knowledge, do not yet exist in the technical literature. If any of these algorithms already exist I would like you to share it with me in an issue.

Random Annealing

A combination between simulated annealing and random search.

Scatter Initialization

Inspired by hyperband optimization.

References

[dto] Scikit-Optimize

License