Skip to main content

Scikit-learn models hyperparameters tuning and features selection, using evolutionary algorithms

Project description

Tests Codecov PythonVersion PyPi Docs

https://github.com/rodrigo-arenas/Sklearn-genetic-opt/blob/master/docs/logo.png?raw=true

Sklearn-genetic-opt

scikit-learn models hyperparameters tuning and feature selection, using evolutionary algorithms.

This is meant to be an alternative to popular methods inside scikit-learn such as Grid Search and Randomized Grid Search for hyperparameters tuning, and from RFE (Recursive Feature Elimination), Select From Model for feature selection.

Table of Contents

  • Sklearn-genetic-opt Overview - Main Features - Demos on Features

  • Installation - Basic Installation - Full Installation with Extras

  • Usage - Hyperparameters Tuning - Feature Selection

  • Documentation - Stable - Latest - Development

  • Changelog

  • Important Links

  • Source Code

  • Contributing

  • Testing

Sklearn-genetic-opt uses evolutionary algorithms from the DEAP (Distributed Evolutionary Algorithms in Python) package to choose the set of hyperparameters that optimizes (max or min) the cross-validation scores, it can be used for both regression and classification problems.

Documentation is available here

Main Features:

  • GASearchCV: Main class of the package for hyperparameters tuning, holds the evolutionary cross-validation optimization routine.

  • GAFeatureSelectionCV: Main class of the package for feature selection.

  • Algorithms: Set of different evolutionary algorithms to use as an optimization procedure.

  • Callbacks: Custom evaluation strategies to generate early stopping rules, logging (into TensorBoard, .pkl files, etc) or your custom logic.

  • Schedulers: Adaptive methods to control learning parameters.

  • Plots: Generate pre-defined plots to understand the optimization process.

  • MLflow: Build-in integration with mlflow to log all the hyperparameters, cv-scores and the fitted models.

Demos on Features:

Visualize the progress of your training:

https://github.com/rodrigo-arenas/Sklearn-genetic-opt/blob/master/docs/images/progress_bar.gif?raw=true

Real-time metrics visualization and comparison across runs:

https://github.com/rodrigo-arenas/Sklearn-genetic-opt/blob/master/docs/images/tensorboard_log.png?raw=true

Sampled distribution of hyperparameters:

https://github.com/rodrigo-arenas/Sklearn-genetic-opt/blob/master/docs/images/density.png?raw=true

Artifacts logging:

https://github.com/rodrigo-arenas/Sklearn-genetic-opt/blob/master/docs/images/mlflow_artifacts_4.png?raw=true

Usage:

Install sklearn-genetic-opt

It’s advised to install sklearn-genetic using a virtual env, inside the env use:

pip install sklearn-genetic-opt

If you want to get all the features, including plotting, tensorboard and mlflow logging capabilities, install all the extra packages:

pip install sklearn-genetic-opt[all]

Example: Hyperparameters Tuning

from sklearn_genetic import GASearchCV
from sklearn_genetic.space import Continuous, Categorical, Integer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score

data = load_digits()
n_samples = len(data.images)
X = data.images.reshape((n_samples, -1))
y = data['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

clf = RandomForestClassifier()

# Defines the possible values to search
param_grid = {'min_weight_fraction_leaf': Continuous(0.01, 0.5, distribution='log-uniform'),
              'bootstrap': Categorical([True, False]),
              'max_depth': Integer(2, 30),
              'max_leaf_nodes': Integer(2, 35),
              'n_estimators': Integer(100, 300)}

# Seed solutions
warm_start_configs = [
           {"min_weight_fraction_leaf": 0.02, "bootstrap": True, "max_depth": None, "n_estimators": 100},
           {"min_weight_fraction_leaf": 0.4, "bootstrap": True, "max_depth": 5, "n_estimators": 200},
    ]

cv = StratifiedKFold(n_splits=3, shuffle=True)

evolved_estimator = GASearchCV(estimator=clf,
                               cv=cv,
                               scoring='accuracy',
                               population_size=20,
                               generations=35,
                               param_grid=param_grid,
                               n_jobs=-1,
                               verbose=True,
                               use_cache=True,
                               warm_start_configs=warm_start_configs,
                               keep_top_k=4)

# Train and optimize the estimator
evolved_estimator.fit(X_train, y_train)
# Best parameters found
print(evolved_estimator.best_params_)
# Use the model fitted with the best parameters
y_predict_ga = evolved_estimator.predict(X_test)
print(accuracy_score(y_test, y_predict_ga))

# Saved metadata for further analysis
print("Stats achieved in each generation: ", evolved_estimator.history)
print("Best k solutions: ", evolved_estimator.hof)

Example: Feature Selection

from sklearn_genetic import GAFeatureSelectionCV, ExponentialAdapter
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
import numpy as np

data = load_iris()
X, y = data["data"], data["target"]

# Add random non-important features
noise = np.random.uniform(5, 10, size=(X.shape[0], 5))
X = np.hstack((X, noise))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0)

clf = SVC(gamma='auto')
mutation_scheduler = ExponentialAdapter(0.8, 0.2, 0.01)
crossover_scheduler = ExponentialAdapter(0.2, 0.8, 0.01)

evolved_estimator = GAFeatureSelectionCV(
    estimator=clf,
    scoring="accuracy",
    population_size=30,
    generations=20,
    mutation_probability=mutation_scheduler,
    crossover_probability=crossover_scheduler,
    n_jobs=-1)

# Train and select the features
evolved_estimator.fit(X_train, y_train)

# Features selected by the algorithm
features = evolved_estimator.support_
print(features)

# Predict only with the subset of selected features
y_predict_ga = evolved_estimator.predict(X_test)
print(accuracy_score(y_test, y_predict_ga))

# Transform the original data to the selected features
X_reduced = evolved_estimator.transform(X_test)

Changelog

See the changelog for notes on the changes of Sklearn-genetic-opt

Source code

You can check the latest development version with the command:

git clone https://github.com/rodrigo-arenas/Sklearn-genetic-opt.git

Install the development dependencies:

pip install -r dev-requirements.txt

Check the latest in-development documentation: https://sklearn-genetic-opt.readthedocs.io/en/latest/

Contributing

Contributions are more than welcome! There are several opportunities on the ongoing project, so please get in touch if you would like to help out. Make sure to check the current issues and also the Contribution guide.

Big thanks to the people who are helping with this project!

Contributors

Testing

After installation, you can launch the test suite from outside the source directory:

pytest sklearn_genetic

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sklearn_genetic_opt-0.11.1.tar.gz (32.3 kB view details)

Uploaded Source

Built Distribution

sklearn_genetic_opt-0.11.1-py3-none-any.whl (35.5 kB view details)

Uploaded Python 3

File details

Details for the file sklearn_genetic_opt-0.11.1.tar.gz.

File metadata

  • Download URL: sklearn_genetic_opt-0.11.1.tar.gz
  • Upload date:
  • Size: 32.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.11.1 requests/2.32.3 setuptools/72.1.0 requests-toolbelt/1.0.0 tqdm/4.66.5 CPython/3.10.14

File hashes

Hashes for sklearn_genetic_opt-0.11.1.tar.gz
Algorithm Hash digest
SHA256 05826e6a0183dc0a11ec343bfdc0b06c01899aca44df422e8a72ccb0ad93b60a
MD5 8f6646784c2785403ffb6f74ef079d61
BLAKE2b-256 c6e76e12e10928dd0dd054aa694f3b10463e3c4d9211f093afb563bc73fc02f3

See more details on using hashes here.

File details

Details for the file sklearn_genetic_opt-0.11.1-py3-none-any.whl.

File metadata

  • Download URL: sklearn_genetic_opt-0.11.1-py3-none-any.whl
  • Upload date:
  • Size: 35.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.11.1 requests/2.32.3 setuptools/72.1.0 requests-toolbelt/1.0.0 tqdm/4.66.5 CPython/3.10.14

File hashes

Hashes for sklearn_genetic_opt-0.11.1-py3-none-any.whl
Algorithm Hash digest
SHA256 96727c871a5f290ba57ef74a6cd6901f28b1e0841516eadac464f8e4c1227fb7
MD5 c28362c96dbe0c9dac1f4123dcce2890
BLAKE2b-256 47031b49e438afc16bdb7271b6d1449e38c111c4a7a0cbe837baef5a681ade0c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page