Basic usage

# Assuming you already have train, test, val sets and a model.
from hypopt import GridSearch
param_grid = [
  {'C': [1, 10, 100], 'kernel': ['linear']},
  {'C': [1, 10, 100], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
 ]
# Grid-search all parameter combinations using a validation set.
gs = GridSearch(model = SVR(), param_grid = param_grid)
gs.fit(X_train, y_train, X_val, y_val)
print('Test Score for Optimized Parameters:', gs.score(X_test, y_test))

Choosing the scoring metric to optimize

The default metric is the the model.score() function, so in the previous example SVR().score() is optimized, which defaults to accuracy.

It’s easy to use a different scoring metric using the scoring parameter in hypopt.GridSearch.fit():

# This will use f1 score as the scoring metric that you optimize.
gs.fit(X_train, y_train, X_val, y_val, scoring='f1')

• For classification, hypopt supports these string-named metrics: ‘accuracy’, ‘brier_score_loss’, ‘average_precision’, ‘f1’, ‘f1_micro’, ‘f1_macro’, ‘f1_weighted’, ‘neg_log_loss’, ‘precision’, ‘recall’, or ‘roc_auc’.

• For regression, hypopt supports: “explained_variance”, “neg_mean_absolute_error”, “neg_mean_squared_error”, “neg_mean_squared_log_error”, “neg_median_absolute_error”, “r2”.

You can also create your own metric your_custom_score_func(y_true, y_pred) by wrapping it into an object using sklearn.metrics.make_scorer like:

from sklearn.metrics import make_scorer
scorer = make_scorer(your_custom_scoring_func)
opt.fit(X_train, y_train, X_val, y_val, scoring=scorer)

Minimal working examples

• Classification minimal working example

• Regression minimal working example

Other Examples including a working example with MNIST

• A simple tutorial of the hypopt package.

• A working example with MNIST.

PyTorch MNIST CNN Example

Check out a PyTorch MNIST CNN wrapped in the above class here. You use any object instantion of this class with hypopt just as you would any scikit-learn model. Another example of a fully compliant class is the LearningWithNoisyLabels() model.

If you don’t wish to write this code yourself, there are existing packages to do this for you. For PyTorch, check out the skorch Python package <https://skorch.readthedocs.io/en/stable/> which will wrap your pytorch model into a scikit-learn compliant model.