lockout 0.1.4

Lockout: Sparse Regularization of Neural Networks

Lockout

Sparsity Inducing Regularization of Fully Connected Neural Networks

Install

pip install lockout [-- upgrade]

Usage

PyTorch installation required.

1. Import Lockout (and PyTorch)

import torch
from lockout import Lockout

2. Neural Network Architecture

Paper

https://arxiv.org/abs/2107.07160

Abstract: Regularized regression and classification procedures attempt to fit a function f(x,ω) of multiple predictor variables x, to data {x_i,y_i}₁^N, based on some loss criterion L(y,f) but adding a constraint P(ω) ≤ t on the joint values of the parameters ω to improve accuracy. While there are efficient methods for finding solutions for all values of t ≥ 0 with some constraints P in the special case that f is a linear function, none exist for non linear functions such as Neural Networks (NN). Here we present a fast algorithm that provides all such solutions for any differentiable function f and loss L, and any constraint P that is an increasing monotone function of the absolute value of each parameter. Applications involving sparsity inducing regularization of arbitrary neural networks are discussed. Empirical results indicate that these sparse solutions are usually superior to their dense counterparts in both accuracy and interpretability. This improvement in accuracy can often make neural networks competitive with, and some times superior to, state of the art methods in the analysis of tabular data.