Preconditioning optimizers.
Project description
precondition
: Preconditioning Optimizers
Installation (note package name is precondition
but pypi distribution name is precondition-opt
):
pip3 install -U precondition-opt
Currently, this contains several preconditioning optimizer implementations. Please refer to the citations below.
Shampoo (distributed_shampoo.py
)
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
Sketchy (distributed_shampoo.py
), logical reference implementation as a branch in Shampoo.
@article{feinberg2023sketchy,
title={Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions},
author={Feinberg, Vladimir and Chen, Xinyi and Sun, Y Jennifer and Anil, Rohan and Hazan, Elad},
journal={arXiv preprint arXiv:2302.03764},
year={2023}
}
SM3 (sm3.py
).
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
This external repository was seeded from existing open-source work available at this google-research repository.
This is not an officially supported Google product.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
precondition-opt-0.3.0.tar.gz
(45.3 kB
view hashes)
Built Distribution
Close
Hashes for precondition_opt-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 850b320d7b67dc9eff099ba7bece83f658039a9281f1f3ae8d12e215c5c29e7e |
|
MD5 | b59a2130f6a2e9d2d8a3629baf343a17 |
|
BLAKE2b-256 | 0ba5be8e8236369271459574f83ca06e0e8e79f64a9186baef801e6b4831dc27 |