AGC Optimizers

A small lib for using adaptive gradient clipping in your optimizer. Currently PyTorch only.

• News
• Introduction
• Comparison
• To Do

News

Sep 15, 2021

• Add AGC use independent from optimizer choice in PyTorch

Sep 14, 2021

• Add AdamW, Adam, SGD and RMSprop with AGC
• Add first comparsion between optimizers with and without AGC based on CIFAR10

Introduction

Brock et al. introduced 2021 a new clipping technique in order to increase stability of large batch training and high learning rates in their Normalizer-Free Networks (NFNet), the adaptive gradient clipping. This clipping method is not implemented in leading frameworks, thus I provide optimizers which are capable of AGC.

Installation

pip install agc_optims

Usage

To be consistent with PyTorch all arguments of the optimizer remain the same as in the standard. Only two parameters are added for the AGC:

• clipping : Hyperparameter for the clipping of the parameter. Default value 1e-2, smaller batch sizes demand a higher clipping parameter
• agc_eps : Term used in AGC to prevent grads clipped to zero, default value 1e-3

Optimizer independent

from torch.optim import Adam
from agc_optims.clipper import AGC

net = Net() # your model

optimizer = Adam(net.parameters(), lr=0.001)
optimizer = AGC(optimizer=optimizer, clipping=0.16)

SGD

from agc_optims.optim import SGD_AGC

net = Net() # your model

optimizer = SGD_AGC(net.parameters(), lr=0.01, momentum=0.9, clipping=0.16)

Adam

from agc_optims.optim import Adam_AGC

net = Net() # your model

optimizer = Adam_AGC(net.parameters(), lr=0.001, weight_decay=1e-4, clipping=0.16)

AdamW

from agc_optims.optim import AdamW_AGC

net = Net() # your model

optimizer = AdamW_AGC(net.parameters(), lr=0.001, weight_decay=1e-4, clipping=0.16)

RMSprop

from agc_optims.optim import RMSprop_AGC

net = Net() # your model

optimizer = RMSprop_AGC(net.parameters(), lr=0.001, clipping=0.16)

Now you can use the optimizer just like their non-AGC counterparts.

Comparison

The following comparison shows that for batch sizes 64 and 128 Adam with AGC performs better than the normal Adam. SGD is unfortunately worse with AGC, but the batch size is also very small compared to the NFNet paper. This requires more comparisons with higher batch sizes and also on other data sets. RMSprop is also better at both batch sizes with AGC than without. The learning rate was left at the default value for all optimizers and the scripts in the performance_tests folder were used as the test environment.

Batch Size 64 - SGD Accuracy on Cifar10	Batch Size 64 - SGD Loss on Cifar10

Batch Size 128 - SGD Accuracy on Cifar10	Batch Size 128 - SGD Loss on Cifar10

Batch Size 64 - Adam Accuracy on Cifar10	Batch Size 64 - Adam Loss on Cifar10

Batch Size 128 - Adam Accuracy on Cifar10	Batch Size 128 - Adam Loss on Cifar10

Batch Size 64 - RMSProp Accuracy on Cifar10	Batch Size 64 - RMSProp Loss on Cifar10

Batch Size 128 - RMSProp Accuracy on Cifar10	Batch Size 128 - RMSProp Loss on Cifar10

As a little treat, I have also compared the speed of the optimizer with and without AGC to see whether this greatly increases training times.

Batch Size 128 - RMSProp Accuracy on Cifar10	Batch Size 128 - RMSProp Loss on Cifar10

To Do

• Add first comparsion based on CIFAR10 with a small CNN
• Add AGC independent from optimizer
• Add comparsion with higher batch sizes (256,512,1024)
• Add tests for each optimizer
• Clipping == 0 no AGC
• Add comparsion based on CIFAR100 with a small CNN
• Add comparsion based on CIFAR10/100 with ResNet
• Add comparsion with ImageNet (I do not have enough GPU-Power currently if someone provides some tests I would be grateful)
• Add all optimizer included in PyTorch
• Support of other frameworks than PyTorch
• Add first comparsion based on CIFAR with a small CNN