Skip to main content

No project description provided

Project description

= TorchTest

A tiny test suite for pytorch based Machine Learning models, inspired by https://github.com/Thenerdstation/mltest/blob/master/mltest/mltest.py[mltest].
Chase Roberts lists out 4 basic tests in his https://medium.com/@keeper6928/mltest-automatically-test-neural-network-models-in-one-function-call-eb6f1fa5019d[medium post] about mltest.
torchtest is sort of a pytorch port of mltest which was written for tensorflow models.


== Tests


[source, python]
----
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

from torchtest import torchtest as tt
----


=== Variables Change

[source, python]
----
inputs = Variable(torch.randn(20, 20))
targets = Variable(torch.randint(0, 2, (20,))).long()
batch = [inputs, targets]
model = nn.Linear(20, 2)

# what are the variables?
print('Our list of parameters', [ np[0] for np in model.named_parameters() ])

# do they change after a training step?
# let's run a train step and see
tt.assert_vars_change(
model=model,
loss_fn=F.cross_entropy,
optim=torch.optim.Adam(model.parameters()),
batch=batch)
----

[source, python]
----
# let's try to break this, so the test fails
params_to_train = [ np[1] for np in model.named_parameters() if np[0] is not 'bias' ]
# run test now
""" FAILURE """
tt.assert_vars_change(
model=model,
loss_fn=F.cross_entropy,
optim=torch.optim.Adam(params_to_train),
batch=batch)

# YES! bias did not change
----


=== Variables Don't Change

[source, python]
----
# What if bias is not supposed to change, by design?
# test to see if bias remains the same after training
tt.assert_vars_same(
model=model,
loss_fn=F.cross_entropy,
optim=torch.optim.Adam(params_to_train),
batch=batch,
params=[('bias', model.bias)]
)
# it does? good. let's move on
----

=== Output Range

[source, python]
----
# NOTE : bias is fixed (not trainable)
optim = torch.optim.Adam(params_to_train)
loss_fn=F.cross_entropy

tt.test_suite(model, loss_fn, optim, batch,
output_range=(-2, 2),
test_output_range=True
)

# seems to work
----

[source, python]
----
# let's tweak the model to fail the test
model.bias = nn.Parameter(2 + torch.randn(2, ))

tt.test_suite(
model,
loss_fn, optim, batch,
output_range=(-1, 1),
test_output_range=True
)

# as expected, it fails; yay!
----

=== NaN Tensors

[source, python]
----
model.bias = nn.Parameter(float('NaN') * torch.randn(2, ))

tt.test_suite(
model,
loss_fn, optim, batch,
test_nan_vals=True
)
----

=== Inf Tensors

[source, python]
----
model.bias = nn.Parameter(float('Inf') * torch.randn(2, ))

tt.test_suite(
model,
loss_fn, optim, batch,
test_inf_vals=True
)
----

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for torchtest, version 0.3
Filename, size File type Python version Upload date Hashes
Filename, size torchtest-0.3.tar.gz (3.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page