About this project

This project is developed by Yngve Mardal Moe and released under an MIT lisence. I am still working out a few things so changes might come rapidly.

Installation guide

Group-lasso requires Python 3.5+, numpy and scikit-learn. To install group-lasso via pip, simply run the command:

pip install group-lasso

Alternatively, you can manually pull this repository and run the setup.py file:

git clone https://github.com/yngvem/group-lasso.git
cd group-lasso
python setup.py

Documentation

You can read the full documentation on readthedocs.

Examples

There are several examples that show usage of the library here.

Further work

1. Fully test with sparse arrays and make examples

2. Make it easier to work with categorical data

3. Poisson regression

Implementation details

The problem is solved using the FISTA optimiser [3] with a gradient-based adaptive restarting scheme [4]. No line search is currently implemented, but I hope to look at that later.

Although fast, the FISTA optimiser does not achieve as low loss values as the significantly slower second order interior point methods. This might, at first glance, seem like a problem. However, it does recover the sparsity patterns of the data, which can be used to train a new model with the given subset of the features.

Also, even though the FISTA optimiser is not meant for stochastic optimisation, it has to my experience not suffered a large fall in performance when the mini batch was large enough. I have therefore implemented mini-batch optimisation using FISTA, and thus been able to fit models based on data with ~500 columns and 10 000 000 rows on my moderately priced laptop.

Finally, we note that since FISTA uses Nesterov acceleration, is not a descent algorithm. We can therefore not expect the loss to decrease monotonically.

References