fseval 2.1.0

Benchmarking framework for Feature Selection algorithms 🚀

fseval

A Feature Selector and Feature Ranker benchmarking library. Neatly integrates with wandb and sklearn. Uses Hydra as a config parser.

Usage

pip install fseval

fseval help:

fseval --help

Now, create a wandb account and login to the CLI. We are now able to run benchmarks 💪🏻. The results will automatically be uploaded to the wandb dashboard.

Run ANOVA F-Value on Iris dataset:

fseval +dataset=iris +estimator@ranker=anova_f_value +estimator@validator=decision_tree

Supported Feature Rankers

A collection of feature rankers are already built-in, which can be used without further configuring. Others need their dependencies installed. List of rankers:

Ranker	Dependency	Command line argument
ANOVA F-Value	<no dep>	estimator@ranker=anova_f_value
Boruta	pip install Boruta	estimator@ranker=boruta
Chi2	<no dep>	estimator@ranker=chi2
Decision Tree	<no dep>	estimator@ranker=decision_tree
FeatBoost	pip install git+https://github.com/dunnkers/FeatBoost.git@support-cloning (ℹ️)	estimator@ranker=featboost
MultiSURF	pip install skrebate	estimator@ranker=multisurf
Mutual Info	<no dep>	estimator@ranker=mutual_info
ReliefF	pip install skrebate	estimator@ranker=relieff
Stability Selection	pip install git+https://github.com/dunnkers/stability-selection.git@master matplotlib (ℹ️)	estimator@ranker=stability_selection
TabNet	pip install pytorch-tabnet	estimator@ranker=tabnet
XGBoost	pip install xgboost	estimator@ranker=xgb
Infinite Selection	pip install git+https://github.com/dunnkers/infinite-selection.git@master (ℹ️)	estimator@ranker=infinite_selection

ℹ️ This library was customized to make it compatible with the fseval pipeline.

If you would like to install simply all dependencies, download the fseval requirements.txt file and run pip install -r requirements.txt.

Wandb support

Wandb can be enabled by using +backend=wandb. It's used to store metrics, but also files. Set any parameter to be passed to wandb.init like so:

fseval callbacks.wandb.project=<your-project-name> callbacks.wandb.group=<run-group>

Runs can be restored as follows:

fseval callbacks.wandb.id=<wandb_run_id> callbacks.wandb.log_metrics=false

→ make sure the rest of the config is the same as the previous run. You can now overwrite tables.

To disable wandb, use:

fseval "~callbacks.wandb"

About

Built by Jeroen Overschie as part of the Masters Thesis (Data Science and Computational Complexity track at the University of Groningen).