fseval

A Feature Selector and Feature Ranker benchmarking library. Neatly integrates with wandb and sklearn. Uses Hydra as a config parser.

Usage

pip install fseval

fseval help:

fseval --help

Now, create a wandb account and login to the CLI. We are now able to run benchmarks 💪🏻. The results will automatically be uploaded to the wandb dashboard.

Run ANOVA F-Value on Iris dataset:

fseval +dataset=iris +estimator@ranker=anova_f_value +estimator@validator=decision_tree

Supported Feature Rankers

A collection of feature rankers are already built-in, which can be used without further configuring. Others need their dependencies installed. List of rankers:

ℹ️ This library was customized to make it compatible with the fseval pipeline.

If you would like to install simply all dependencies, download the fseval requirements.txt file and run pip install -r requirements.txt.

Wandb support

Wandb can be enabled by using +backend=wandb. It's used to store metrics, but also files. Set any parameter to be passed to wandb.init like so:

fseval callbacks.wandb.project=<your-project-name> callbacks.wandb.group=<run-group>

Runs can be restored as follows:

fseval callbacks.wandb.id=<wandb_run_id> callbacks.wandb.log_metrics=false

→ make sure the rest of the config is the same as the previous run. You can now overwrite tables.

To disable wandb, use:

fseval "~callbacks.wandb"

About

Built by Jeroen Overschie as part of the Masters Thesis (Data Science and Computational Complexity track at the University of Groningen).