hyperactive 2.2.1

A hyperparameter optimization toolbox for convenient and fast prototyping

---

A hyperparameter optimization and meta-learning toolbox for convenient and fast prototyping of machine-learning models.

Master status:
Dev status:
Code quality:
Latest versions:

Hyperactive is primarly a hyperparameter optimization toolkit, that aims to simplify the model-selection and -tuning process. You can use any machine- or deep-learning package and it is not necessary to learn new syntax. Hyperactive offers high versatility in model optimization because of two characteristics:

• You can define any kind of model in the objective function. It just has to return a score/metric that gets maximized.
• The search space accepts not just int, float or str as data types but even functions, classes or any python objects.

For more information, visualization and details about the API check out the
website

---

Main features • Installation • Roadmap • License

---

Main features

• Thoroughly tested code base
• Compatible with any python machine-learning framework
• Optimize:
  • Anything from simple models
    to complex machine-learning-pipelines
  • Multi-level ensembles
  • Deep neural network architecture
  • Other optimization techniques (meta-optimization)
  • Or any function you can specify with this API
• Utilize state of the art optimization techniques like:
  • Simulated annealing
  • Evolution strategy
  • Bayesian optimization
• High performance: Optimizer time is neglectable for most models
• Choose from a variety of different optimization extensions to improve the optimization

Optimization Techniques	Tested and Supported Packages	Optimization Extensions
Local Search: Hill Climbing Stochastic Hill Climbing Tabu Search Random Methods: Random Search Random Restart Hill Climbing Random Annealing [*] Markov Chain Monte Carlo: Simulated Annealing Stochastic Tunneling Parallel Tempering Population Methods: Particle Swarm Optimizer Evolution Strategy Sequential Methods: Bayesian Optimization Tree of Parzen Estimators	Machine Learning: Scikit-learn XGBoost LightGBM CatBoost RGF Mlxtend Deep Learning: Tensorflow Keras Distribution: Multiprocessing Ray	Position Initialization: Scatter-Initialization [*] Warm-start Resource Allocation: Memory Short term Long term

Installation

The most recent version of Hyperactive is available on PyPi:

pip install hyperactive

Roadmap

v2.0.0:heavy_check_mark:

• Change API
• Ray integration

v2.1.0:heavy_check_mark:

• Save memory of evaluations for later runs (long term memory)
• Warm start sequence based optimizers with long term memory
• Gaussian process regressors from various packages (gpy, sklearn, GPflow, ...) via wrapper

v2.2.0:heavy_check_mark:

• Add basic dataset meta-features to long term memory
• Add helper-functions for memory
  • connect two different model/dataset hashes
  • split two different model/dataset hashes
  • delete memory of model/dataset
  • return best known model for dataset
  • return search space for best model
  • return best parameter for best model

v2.3.0

• Tree-structured Parzen Estimator
• Multiple experimental SBOM
• Meta-Optimization of all optimizers

v2.4.0

• Spiral optimization
• Downhill-Simplex-Method
• Differential evolution

v2.5.0

• Helper-functions for early stopping
• Helper-functions for dataset approximation

Experimental algorithms

The following algorithms are of my own design and, to my knowledge, do not yet exist in the technical literature. If any of these algorithms already exist I would like you to share it with me in an issue.

Random Annealing

A combination between simulated annealing and random search.

Scatter Initialization

Inspired by hyperband optimization.

References

[1] Proxy Datasets for Training Convolutional Neural Networks

License