Preconditioning optimizers.
Project description
precondition
: Preconditioning Optimizers
Installation (note package name is precondition
but pypi distribution name is precondition-opt
):
pip3 install -U precondition-opt
Currently, this contains several preconditioning optimizer implementations. Please refer to the citations below.
Shampoo (distributed_shampoo.py
)
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
Sketchy (distributed_shampoo.py
), logical reference implementation as a branch in Shampoo.
@article{feinberg2023sketchy,
title={Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions},
author={Feinberg, Vladimir and Chen, Xinyi and Sun, Y Jennifer and Anil, Rohan and Hazan, Elad},
journal={arXiv preprint arXiv:2302.03764},
year={2023}
}
SM3 (sm3.py
).
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
This external repository was seeded from existing open-source work available at this google-research repository.
This is not an officially supported Google product.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
precondition-opt-0.2.0.tar.gz
(45.3 kB
view hashes)
Built Distribution
Close
Hashes for precondition_opt-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25e47c2bf96c6eee70bc434e2c90a338f7403441c2dd35ebb21157cf400ae605 |
|
MD5 | 630e35f675276364ad1ab36e1f830998 |
|
BLAKE2b-256 | 52385c4b788529a8e8e1e8733ba5676b4bfb004198e76ca33a0c5b5add923e76 |