curvlinops-for-pytorch 1.1.0

scipy Linear operator implementations of the GGN and Hessian in PyTorch

scipy linear operators of deep learning matrices in PyTorch

This library implements scipy.sparse.linalg.LinearOperators for deep learning matrices, such as

• the Hessian
• the Fisher/generalized Gauss-Newton (GGN)
• the Monte-Carlo approximated Fisher
• the uncentered gradient covariance (aka empirical Fisher)

Matrix-vector products are carried out in PyTorch, i.e. potentially on a GPU. The library supports defining these matrices not only on a mini-batch, but on data sets (looping over batches during a matvec operation).

You can plug these linear operators into scipy, while carrying out the heavy lifting (matrix-vector multiplies) in PyTorch on GPU. My favorite example for such a routine is scipy.sparse.linalg.eigsh that lets you compute a subset of eigenpairs.

• Documentation: https://curvlinops.readthedocs.io/en/latest/

• Bug reports & feature requests: https://github.com/f-dangel/curvlinops/issues

Installation

pip install curvlinops-for-pytorch

Examples

• Basic usage

Future ideas

Other features that could be supported in the future include:

• Other matrices

  • the centered gradient covariance
  • terms of the hierarchical GGN decomposition

• Block-diagonal approximations (via param_groups)

Logo mage credits

• SciPy logo: Unknown, CC BY-SA 4.0, via Wikimedia Commons
• PyTorch logo: https://github.com/soumith, CC BY-SA 4.0, via Wikimedia Commons