Adas: Adaptive Scheduling of Stochastic Gradients

Status

Table of Contents

• Adas: Adaptive Scheduling of Stochastic Gradients
  • Status
  • Introduction
    • License
    • Citing Adas
    • Empirical Classification Results on CIFAR10 and CIFAR100
    • QC Metrics
  • Requirements
    • Software/Hardware
    • Computational Overhead
    • Installation
    • Usage
    • Common Issues (running list)
  • TODO
    • Pytest

Introduction

Adas is an adaptive optimizer for scheduling the learning rate in training Convolutional Neural Networks (CNN)

• Adas exhibits the rapid minimization characteristics that adaptive optimizers like AdaM are favoured for
• Adas exhibits generalization (low testing loss) characteristics on par with SGD based optimizers, improving on the poor generalization characteristics of adaptive optimizers
• Adas introduces no computational overhead over adaptive optimizers (see experimental results)
• In addition to optimization, Adas introduces new probing metrics for CNN layer evaulation (quality metrics)

This repository contains a PyTorch implementation of the Adas learning rate scheduler algorithm as well as the Knowledge Gain and Mapping Condition metrics.

Visit the paper branch to see the paper-related code. You can use that code to replicate experiments from the paper.

License

Adas is released under the MIT License (refer to the LICENSE file for more information)

Permissions	Conditions	Limitations
Commerical use	License and Copyright Notice	Liability
Distribution		Warranty
Modification
Private Use

Citing Adas

@article{hosseini2020adas,
  title={Adas: Adaptive Scheduling of Stochastic Gradients},
  author={Hosseini, Mahdi S and Plataniotis, Konstantinos N},
  journal={arXiv preprint arXiv:2006.06587},
  year={2020}
}

Empirical Classification Results on CIFAR10, CIFAR100 and Tiny-ImageNet-200

Figure 1: Training performance using different optimizers across three datasets and two CNNs

Table 1: Image classification performance (test accuracy) with fixed budget epoch of ResNet34 training

QC Metrics

Please refer to QC on Wiki for more information on two metrics of knowledge gain and mapping condition for monitoring training quality of CNNs

Requirements

Software/Hardware

We use Python 3.7.

Please refer to Requirements on Wiki for complete guideline.

Computational Overhead

Adas introduces no overhead (very minimal) over adaptive optimizers e.g. all mSGD+StepLR, mSGD+Adas, AdaM consume 40~43 sec/epoch to train ResNet34/CIFAR10 using the same PC/GPU platform

Installation

1. You can install Adas directly from PyPi using `pip install adas', or clone this repository and install from source.
2. You can also download the files in src/adas into your local code base and use them directly. Note that you will probably need to modify the imports to be consistent with however you perform imports in your codebase.

All source code can be found in src/adas

For more information, also refer to Installation on Wiki

Usage

The use Adas, simply import the Adas(torch.optim.optimier.Optimizer) class and use it as follows:

from adas import Adas

optimizer = Adas(params=list(model.parameters()),
                 lr: float = ???,
                 beta: float = 0.8
                 step_size: int = None,
                 gamma: float = 1,
                 momentum: float = 0,
                 dampening: float = 0,
                 weight_decay: float = 0,
                 nesterov: bool = False):
...
for epoch in epochs:
    for batch in train_dataset:
        ...
        loss.backward()
        optimizer.step()
    optimizer.epoch_step(epoch)

Note, optipmizer.epoch_step() is just to be called at the end of each epoch.

Common Issues (running list)

• None :)

TODO

• Add medical imaging datasets (e.g. digital pathology, xray, and ct scans)
• Extension of Adas to Deep Neural Networks

Pytest

Note the following:

• Our Pytests write/download data/files etc. to /tmp, so if you don't have a /tmp folder (i.e. you're on Windows), then correct this if you wish to run the tests yourself