ViViT: Curvature access through the generalized Gauss-Newton\'s low-rank structure
Project description
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
ViViT is a collection of numerical tricks to efficiently access curvature from the generalized Gauss-Newton (GGN) matrix based on its low-rank structure. Provided functionality includes computing
- GGN eigenvalues (basic example)
- GGN eigenpairs (eigenvalues + eigenvector, basic example)
- 1ˢᵗ- and 2ⁿᵈ-order directional derivatives along GGN eigenvectors (basic example)
- Directionally damped Newton steps (basic example)
These operations can also further approximate the GGN to reduce cost via sub-sampling, Monte-Carlo approximation, and block-diagonal approximation.
- Documentation: https://vivit.readthedocs.io/en/latest/
- Bug reports & feature requests: https://github.com/f-dangel/vivit/issues
How does it work? ViViT uses and extends BackPACK for PyTorch. The described functionality is realized through a combination of existing and new BackPACK extensions and hooks into its backpropagation.
Installation
pip install vivit-for-pytorch
Examples
Basic and advanced demos can be found in the documentation.
How to cite
If you are using ViViT, consider citing the paper
@misc{dangel2022vivit,
title={{ViViT}: Curvature access through the generalized Gauss-Newton's low-rank structure},
author={Felix Dangel and Lukas Tatzel and Philipp Hennig},
year={2022},
eprint={2106.02624},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for vivit_for_pytorch-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6e3b5d66f09176aa5ef699811ced3e455dee03ef4e27b6dc8aca050b1d8ac39 |
|
MD5 | 0c73c5ce979149ed4755e691c54b03d9 |
|
BLAKE2b-256 | eb0d3930bf4c3a2bb5c8b02ccc99c5b52eca9b975496ba5218f95c65f9bf8da5 |