curvlinops-for-pytorch 1.0.0

scipy Linear operator implementations of the GGN and Hessian in PyTorch

scipy linear operators of deep learning matrices in PyTorch

This library implements scipy.sparse.linalg.LinearOperators for deep learning matrices, such as

• the Hessian
• the Fisher/generalized Gauss-Newton (GGN)

Matrix-vector products are carried out in PyTorch, i.e. potentially on a GPU. The library supports defining these matrices not only on a mini-batch, but on data sets (looping over batches during a matvec operation).

You can plug these linear operators into scipy, while carrying out the heavy lifting (matrix-vector multiplies) in PyTorch on GPU. My favorite example for such a routine is scipy.sparse.linalg.eigsh that lets you compute a subset of eigenpairs.

• Documentation: https://curvlinops.readthedocs.io/en/latest/

• Bug reports & feature requests: https://github.com/f-dangel/curvlinops/issues

Installation

pip install curvlinops-for-pytorch

Examples

• Basic usage

Future ideas

Other features that could be supported in the future include:

• Other matrices

  • the un-centered gradient covariance (aka empirical Fisher)
  • the centered gradient covariance
  • terms of the hierarchical GGN decomposition

• Block-diagonal approximations (via param_groups)

• Inverse matrix-vector products by solving a linear system via conjugate gradients

  • This could allow computing generalization metrics like the Takeuchi Information Criterion (TIC), using inverse matrix-vector products in combination with Hutchinson trace estimation

Logo mage credits

• SciPy logo: Unknown, CC BY-SA 4.0, via Wikimedia Commons
• PyTorch logo: https://github.com/soumith, CC BY-SA 4.0, via Wikimedia Commons