Adam-atan2 for Pytorch
Project description
Adam-atan2 - Pytorch
Implementation of the proposed Adam-atan2 optimizer in Pytorch
A multi-million dollar paper out of google deepmind proposes a small change to Adam update rule (using atan2
) to remove the epsilon altogether for numerical stability and scale invariance
Install
$ pip install adam-atan2-pytorch
Usage
import torch
from torch import nn
# toy model
model = nn.Linear(10, 1)
# import AdamAtan2 and instantiate with parameters
from adam_atan2_pytorch import AdamAtan2
opt = AdamAtan2(model.parameters(), lr = 1e-4)
# forward and backwards
for _ in range(100):
loss = model(torch.randn(10))
loss.backward()
# optimizer step
opt.step()
opt.zero_grad()
Citations
@inproceedings{Everett2024ScalingEA,
title = {Scaling Exponents Across Parameterizations and Optimizers},
author = {Katie Everett and Lechao Xiao and Mitchell Wortsman and Alex Alemi and Roman Novak and Peter J. Liu and Izzeddin Gur and Jascha Narain Sohl-Dickstein and Leslie Pack Kaelbling and Jaehoon Lee and Jeffrey Pennington},
year = {2024},
url = {https://api.semanticscholar.org/CorpusID:271051056}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
adam_atan2_pytorch-0.0.10.tar.gz
(418.0 kB
view hashes)
Built Distribution
Close
Hashes for adam_atan2_pytorch-0.0.10.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09452956cc9d6f9fa7c7cc2fe5234070f995e0571b5e9fd2cc514a772160d987 |
|
MD5 | 76069f969539957588e2ab286a83b1ac |
|
BLAKE2b-256 | ed766457daa098aaa4e31a814d6800452f5188f137afddf3a74ac6b00c6c2ef5 |
Close
Hashes for adam_atan2_pytorch-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1e3a4dd26f68b2506010d71285f5f7dd02f400140ca883b20376d3343d61103 |
|
MD5 | 36c232c712a5c982de3fffd3931cd67e |
|
BLAKE2b-256 | cecb169e68dc7979e35bf2d696c9867658f7e34c3b073ee8b0a47a4d26dbf0c9 |