pytorch-lamb

No project description provided

Project description

Implementation of https://arxiv.org/abs/1904.00962 for large batch, large learning rate training.

The paper doesn't specify clamp values for ϕ, so I use 10.

Bonus: TensorboardX logging (example below).

git clone git@github.com:cybertronai/pytorch-lamb.git
cd pytorch-lamb
pip install -e .
python test_lamb.py
tensorboard --logdir=runs

At --lr=.02, the Adam optimizer is unable to train.

Red: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=adam

Blue: python test_lamb.py --batch-size=512 --lr=.02 --wd=.01 --log-interval=30 --optimizer=lamb

This version

1.0.0

Jun 16, 2019

0.0.0

May 2, 2019

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Uploaded Jun 16, 2019 Source

Uploaded Jun 16, 2019 Python 3

Details for the file pytorch_lamb-1.0.0.tar.gz.

Download URL: pytorch_lamb-1.0.0.tar.gz
Upload date: Jun 16, 2019
Size: 3.1 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.1

Hashes for pytorch_lamb-1.0.0.tar.gz
Algorithm	Hash digest
SHA256	`adf871c5e95ca70c8d3cffcb03408064af6e6fc29ddcc67e9efaf3a7caa0a288`
MD5	`fe3597ab2ec4e02ad07f506d4db6c52a`
BLAKE2b-256	`fbd50627bb028802787f0cc501b0c3d71bd6ab1fa3e8d08d42ff111745d4dd75`

Details for the file pytorch_lamb-1.0.0-py3-none-any.whl.

Download URL: pytorch_lamb-1.0.0-py3-none-any.whl
Upload date: Jun 16, 2019
Size: 4.4 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.1

Hashes for pytorch_lamb-1.0.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`0cdc4dffe0b402ee55f482466ebab5b25332a1036cef595d5c733bac0a37be71`
MD5	`0a4e996785f37d35ddb651714c98692d`
BLAKE2b-256	`43983bce14a319317a2856db722f2542d329baf42845fa53563d0d749c5a2d40`