pytorch-lamb

No project description provided

Project description

Implementation of https://arxiv.org/abs/1904.00962 for large batch, large learning rate training.

The paper doesn't specify clamp values for ϕ, so I use 10.

Bonus: TensorboardX logging (example below).

git clone git@github.com:cybertronai/pytorch-lamb.git
cd pytorch-lamb
pip install -e .
python test_lamb.py
tensorboard --logdir=runs

At --lr=.02, the Adam optimizer is unable to train.

Red: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=adam

Blue: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=lamb

This version

1.0.0

Jun 16, 2019

0.0.0

May 2, 2019

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Uploaded Jun 16, 2019 Source

Uploaded Jun 16, 2019 Python 3

Hashes for pytorch_lamb-1.0.0.tar.gz
Algorithm	Hash digest
SHA256	`adf871c5e95ca70c8d3cffcb03408064af6e6fc29ddcc67e9efaf3a7caa0a288`
MD5	`fe3597ab2ec4e02ad07f506d4db6c52a`
BLAKE2b-256	`fbd50627bb028802787f0cc501b0c3d71bd6ab1fa3e8d08d42ff111745d4dd75`

Hashes for pytorch_lamb-1.0.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`0cdc4dffe0b402ee55f482466ebab5b25332a1036cef595d5c733bac0a37be71`
MD5	`0a4e996785f37d35ddb651714c98692d`
BLAKE2b-256	`43983bce14a319317a2856db722f2542d329baf42845fa53563d0d749c5a2d40`