HypONIC -- Hyperparameter Optimization with Nature Inspired Computing

HypONIC is a hyperparameter optimization library that uses various nature inspired computing algorithms to optimize the hyperparameters of machine learning models. The library provides a simple interface for sklearn models.

Documentation: https://hyponic.readthedocs.io/en/latest/

Library Structure

|-- hyponic/
|   |-- hyponic.py
|   |-- space.py
|   |
|   |-- metrics/
|   |   |-- classification.py
|   |   |-- regression.py
|   |   |-- decorators.py
|   |
|   |-- optimizers/
|   |   |-- base_optimizer.py
|   |   |
|   |   |-- swarm_based/
|   |   |   |-- ABC.py
|   |   |   |-- ACO.py
|   |   |   |-- PSO.py
|   |   |
|   |   |-- physics_based/
|   |   |   |-- SA.py
|   |   |
|   |   |-- genetic_based/
|   |       |-- GA.py
|   |
|   |-- space/
|   |   |-- encoders.py (planned)
|   |   |-- mappers.py  (planned)
|   |
|   |-- utils/
|       |-- history.py
|       |-- problem_identifier.py
|-- examples/
    |-- datasets/

Installation

Install from PyPI

pip install hyponic

Minimal Example

This is a minimal example for tuning hyperparameters of the SVC from sklearn. The default optimization is Inertia Weight Particle Swarm Optimization (IWPSO), as it has high applicability and has shown fast convergence.

# 1. Import of the model, dataset, metric and optimizer
#    with the algorithm of choice
from sklearn.svm import SVC
from sklearn.datasets import load_wine

from hyponic.hyponic import HypONIC

#    ...in this case, log_loss and Inertia Weight PSO
from hyponic.metrics.classification import log_loss
from hyponic.optimizers.swarm_based.PSO import IWPSO

# 2. Creating the model and loading the dataset
model = SVC()
X, y = load_wine(return_X_y=True)

# 3. Defining our problem
hyperparams = {
    "C": (0.01, 1),
    "kernel": ["linear", "poly", "rbf", "sigmoid"],
    "degree": [2, 3, 4, 5],
}

# 4. Parameterizing the optimizer
optimizer_kwargs = {
    "epoch": 10,
    "population_size": 10,
}

# 5. Passing all to the optimizer
hyponic = HypONIC(
    model,
    X, y,
    log_loss,
    optimizer=IWPSO,
    **optimizer_kwargs
)

# 6. Solving for the given problem
hyponic.optimize(hyperparams, verbose=True)
print(hyponic.get_optimized_parameters())
print(hyponic.get_optimized_metric())
print(hyponic.get_optimized_model())

Implementation details

HypONIC library follows a common high-level approach by providing a unified interface for applying an optimization algorithm of choice to the parameters of a machine learning model (based on the BaseEstimator from the sklearn). Evaluation of the optimization process is done by a metric of choice.

Metrics

HypONIC provides a list of classical evaluation metrics:

• Mean Squared Error
• Mean Absolute Error
• Root Mean Square Error
• Huber Loss
• Log Loss
• Binary Cross entropy
• Precision Score
• Accuracy Score
• Recall Score
• F1 Score
• R2 Score
• ...and many others.

HypONIC library provides decorators for annotating custom metrics: @minimize_metric, @maximize_metric, and @add_metric_aliases.

Use case for the first two decorators is to annotate the metric function with the optimization goal, i.e. let's say we want to add a custom metric function custom_metric to the library, and we want to minimize it. To do so, we annotate the function with the @minimize_metric decorator:

from hyponic.hyponic import HypONIC
from hyponic.optimizers.swarm_based.PSO import IWPSO
from hyponic.metrics.decorators import minimize_metric

model = ...
X, y = ..., ...
optimizer_kwargs = ...

# -- HERE IS THE DECORATOR --
@minimize_metric
def custom_metric(y_true, y_pred):
    ...

hyponic = HypONIC(model, X, y, custom_metric, optimizer=IWPSO, **optimizer_kwargs)

Under the hood, the decorator adds a minimize attribute to the function, which is later used by the optimizer to determine the optimization goal.

Not only that, but the decorator also adds the function to the list of available metrics, so that it can be used as a string literal in the HypONIC constructor, i.e.

from hyponic.hyponic import HypONIC
from hyponic.optimizers.swarm_based.PSO import IWPSO
from hyponic.metrics.decorators import minimize_metric

model = ...
X, y = ..., ...
optimizer_kwargs = ...

@minimize_metric
def custom_metric(y_true, y_pred):
    ...

hyponic = HypONIC(model, X, y, "custom_metric", optimizer=IWPSO, **optimizer_kwargs)

For the standard metrics, the decorators are already applied, so that they can be used as string literals in the HypONIC constructor without the need to import them from the hyponic.metrics module.

By default, function aliases are generated from the function name, however, it is possible to add custom aliases using the @add_metric_aliases decorator:

from hyponic.hyponic import HypONIC
from hyponic.optimizers.swarm_based.PSO import IWPSO
from hyponic.metrics.decorators import minimize_metric, add_metric_aliases

model = ...
X, y = ..., ...
optimizer_kwargs = ...

hyperparams = ...

@minimize_metric
@add_metric_aliases("custom_metric", "short_alias")
def custom_metric(y_true, y_pred):
    ...

hyponic = HypONIC(model, X, y, "short_alias", optimizer=IWPSO, **optimizer_kwargs)
hyponic.optimize(hyperparams, verbose=True)

Output that this code produces will show that the short_alias, used as a metric alias, resolves to the custom_metric.

Hyperparameter Space

HypONIC library supports mixed space of hyperparameters, meaning that one optimization space can have both dimensions of discrete and continuous parameters. The notation for each space is the following:

hyperparams = {
    "min_impurity_decrease": (0.0, 0.9),              # Continuous space -- (lower bound, upper bound)
    "min_samples_split": [2, 3, 4, 5, 6],             # Discrete space   -- a list of values of any type
    "criterion": ["absolute_error", "squared_error"]  # Discrete space   -- a list of values of any type
}

For the discrete space an automatic uniform mapping* to the continuous space is provided if needed. Discrete space of non-numerical values is encoded first and then uniformly mapped to the continuous space.

*is subject to change

Features

HypONIC supports different nature inspired computing algorithms. Please note that the library is currently under development and may contain a significant number of bugs and errors. All algorithms are subject to further improvement.

The following algorithms are currently implemented or are planned to be implemented:

Swarm-based:

• • Particle Swarm Optimization (PSO)
• • Inertia Weight PSO (IWPSO)
• • Ant Colony Optimization (ACO)
• • Artificial Bee Colony (ABC)
• • Grey Wolf Optimizer (GWO)
• • Cuckoo Search (CS)
• • Firefly Algorithm (FA)

Physics-based:

• • Simulated Annealing (SA)

Genetic-based:

• • Genetic Algorithm (GA)

The following features are currently implemented or are planned to be implemented:

• • Early stopping based on the number of epoch without improvement
• • History of optimization
• • Parallelization
• • Visualization of the history
• • Visualization of the optimization process
• • Customizable stopping criteria
• • Partial fit
• • Different population initializations
• • Identifying the problem type (regression, classification)
• • Support for pytorch or keras models