torchgel 0.8.0

PyTorch implementation of group elastic net

# torch-gel

[![Travis][travis-badge]][travis]
[![PyPI][pypi-badge]][pypi]
[![PyPi - Python Version][pypi-version-badge]][pypi]
[![GitHub license][license-badge]][license]
[![Code style: black][black-badge]][black]

[travis-badge]: https://img.shields.io/travis/jayanthkoushik/torch-gel.svg?style=for-the-badge
[travis]: https://travis-ci.com/jayanthkoushik/torch-gel
[pypi-badge]: https://img.shields.io/pypi/v/torchgel.svg?style=for-the-badge
[pypi-version-badge]: https://img.shields.io/pypi/pyversions/torchgel.svg?style=for-the-badge
[pypi]: https://pypi.org/project/torchgel/
[license-badge]: https://img.shields.io/github/license/jayanthkoushik/torch-gel.svg?style=for-the-badge
[license]: https://github.com/jayanthkoushik/torch-gel/blob/master/LICENSE.txt
[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge
[black]: https://github.com/ambv/black

This package provides PyTorch implementations to solve the group elastic net
problem. Let _A_j_ (_j = 1 … p_) be feature matrices of sizes _m ⨉
n_j_ (_m_ is the number of samples, and _n_j_ is the number
of features in the _j_^th group), and let _y_ be an _m ⨉ 1_ vector of
the responses. Group elastic net finds coefficients _β_j_, and a bias
_β₀_ that solve the optimization problem

> min __{β₀, …, β_p}_
> _½ ║y - β₀ - ∑ A_j β_j║²_
> + _m ∑ √n_j (λ₁║β_j║_
> _+ λ₂║β_j║²)._

Here _λ₁_ and _λ₂_ are scalar coefficients that control
the amount of 2-norm and squared 2-norm regularization. This 2-norm
regularization encourages sparsity at the group level; entire _β_j_
might become 0. The squared 2-norm regularization is in similar spirit to
elastic net, and addresses some of the issues of lasso. Note that group elastic
net includes as special cases group lasso (_λ₂ = 0_), ridge
regression (_λ₁ = 0_), elastic net (each _n_j = 1_), and
lasso (each _n_j = 1_ and _λ₂ = 0_). The optimization
problem is convex, and can be solved efficiently. This package provides two
implementations; one based on proximal gradient descent, and one based on
coordinate descent.

## Installation
Install with `pip`

```bash
pip install torchgel
```

`tqdm` (for progress bars) is pulled in as a dependency. PyTorch (`v0.4.*`) is
also needed, and needs to be installed manually. Refer to the [PyTorch
website](<http://pytorch.org>) for instructions.

## Solving Single Instances
The modules `gel.gelfista` and `gel.gelcd` provide implementations based on
proximal gradient descent and coordinate descent respectively. Both have similar
interfaces, and expose two main public functions: `make_A` and `gel_solve`. The
feature matrices should be stored in a list (say `As`) as PyTorch tensor
matrices, and the responses should be stored in a PyTorch vector (say `y`).
Additionally, the sizes of the groups (_n_j_) should be stored in a
vector (say `ns`). First use the `make_A` function to convert the feature
matrices into a suitable format:

```python
A = make_A(As, ns)
```

Then pass `A`, `y` and other required arguments to `gel_solve`. The general
interface is::

```python
b_0, B = gel_solve(A, y, l_1, l_2, ns, **kwargs)
```

`l_1` and `l_2` are floats representing _λ₁_ and _λ₂_
respectively. The method returns a float `b_0` representing the bias and a
PyTorch matrix `B` holding the other coefficients. `B` has size _p ⨉_
max_{_j_} _n_j_ with suitable zero padding. The following
sections cover additional details for the specific implementations.

### Proximal Gradient Descent (FISTA)
The `gel.gelfista` module contains a proximal gradient descent implementation.
It's usage is just as described in the template above. Refer to the docstring
for `gel.gelfista.gel_solve` for details about the other arguments.

### Coordinate Descent
The `gel.gelcd` module contains a coordinate descent implementation. Its usage
is a bit more involved than the FISTA implementation. Coordinate descent
iteratively solves single blocks (each corresponding to a single
_β_j_). There are multiple solvers provided to solve the individual
blocks. These are the `gel.gelcd.block_solve_*` functions. Refer to their
docstrings for details about their arguments. `gel.gelcd.gel_solve` requires
passing a block solve function and its arguments (as a dictionary). Refer to
its docstring for further details.

## Solution Paths
`gel.gelpaths` provides a wrapper function `gel_paths` to solve the group
elastic net problem for multiple values of the regularization coefficients. It
implements a two-stage process. For a given _λ₁_ and _λ₂_,
first the group elastic net problem is solved and the feature blocks with
non-zero coefficients is extracted (the support). Then ridge regression models
are learned for each of several provided regularization values. The final model
is summarized using an arbitrary provided summary function, and the summary for
each combination of the regularization values is returned as a dictionary. The
docstring contains more details. The module contains another useful function
`ridge_paths` which can efficiently solve ridge regression for multiple
regularization values.

## Citation
If you find this code useful in your research, please cite

```
@misc{koushik2017torchgel,
author = {Koushik, Jayanth},
title = {torch-gel},
year = {2017},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/jayanthkoushik/torch-gel}},
}
```