gradients 0.0.3

Gradient Checker for Custom built PyTorch Models

Build your deep learning models with confidence

Gradients provide a self consistency test function to perform gradient checking on your deep learning models. It uses centered finite difference approximation method to check the difference between analytical and numerical gradients and report if the check fails on any parameters of your model. Currently the library supports only PyTorch models built with custom layers, custom loss functions, activation functions and any neural network function subclassing AutoGrad.

Installation

pip install gradients

Package Overview

Optimizing deep learning models is a two step process:

1. Compute gradients with respect to parameters

2. Update the parameters given the gradients

In PyTorch, step 1 is done by the type-based automatic differentiation system torch.nn.autograd and 2 by the package implementing optimization algorithms torch.optim. Using them, we can develop fully customized deep learning models with torch.nn.Module and test them using Gradient as follows;

Activation function with backward

class MySigmoid(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        output = 1/(1+torch.exp(-input))
        ctx.save_for_backward(output)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        return grad_output*input*(1-input)

Loss function with backward

class MSELoss(torch.autograd.Function):

    @staticmethod
    def forward(ctx, y_pred, y):
        ctx.save_for_backward(y_pred, y)
        return ((y_pred-y)**2).sum()/y_pred.shape[0]

    @staticmethod
    def backward(ctx, grad_output):
        y_pred, y = ctx.saved_tensors
        grad_input = 2 * (y_pred-y)/y_pred.shape[0]
        return grad_input, None

Pytorch Model

class MyModel(torch.nn.Module):
    def __init__(self,D_in, D_out):
        super(MyModel,self).__init__()
        self.w1 = torch.nn.Parameter(torch.randn(D_in, D_out), requires_grad=True)
        self.sigmoid = MySigmoid.apply
    def forward(self,x):
        y_pred = self.sigmoid(x.mm(self.w1))
        return y_pred

Check your implementation using Gradient

import torch
from gradients import Gradient

N, D_in, D_out = 10, 4, 3

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Construct model by instantiating the class defined above
mymodel = MyModel(D_in, D_out)
criterion = MSELoss.apply

# Test custom build model
Gradient(mymodel,x,y,criterion,eps=1e-8)

Citing Gradients

@software{nambusubramaniyan_saranraj_2021_5176243,
  author       = {Nambusubramaniyan, Saranraj},
  title        = {gradients},
  month        = aug,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.5176243},
  url          = {https://doi.org/10.5281/zenodo.5176243}
}```