Skip to main content

Tools for Robustness

Project description

Install via pip: pip install robustness

Read the docs: https://robustness.readthedocs.io/en/latest/index.html

robustness is a package we (students in the MadryLab) created to make training, evaluating, and exploring neural networks flexible and easy. We use it in almost all of our projects (whether they involve adversarial training or not!) and it will be a dependency in many of our upcoming code releases. A few projects using the library include:

We demonstrate how to use the library in a set of walkthroughs and our API reference. Functionality provided by the library includes:

python -m robustness.main --dataset cifar --data /path/to/cifar \
   --adv-train 0 --arch resnet18 --out-dir /logs/checkpoints/dir/
  • Performing input manipulation using robust (or standard) models—this includes making adversarial examples, inverting representations, feature visualization, etc. The library offers a variety of optimization options (e.g. choice between real/estimated gradients, Fourier/pixel basis, custom loss functions etc.), and is easily extendable.

import torch as ch
from robustness.datasets import CIFAR
from robustness.model_utils import make_and_restore_model

ds = CIFAR('/path/to/cifar')
model, _ = make_and_restore_model(arch='resnet50', dataset=ds,
             resume_path='/path/to/model', state_dict_path='model')
model.eval()
attack_kwargs = {
   'constraint': 'inf', # L-inf PGD
   'eps': 0.05, # Epsilon constraint (L-inf norm)
   'step_size': 0.01, # Learning rate for PGD
   'iterations': 100, # Number of PGD steps
   'targeted': True # Targeted attack
   'custom_loss': None # Use default cross-entropy loss
}

_, test_loader = ds.make_loaders(workers=0, batch_size=10)
im, label = next(iter(test_loader))
target_label = (label + ch.randint_like(label, high=9)) % 10
adv_out, adv_im = model(im, target_label, make_adv, **attack_kwargs)
  • Importing robustness as a package, which allows for easy training of neural networks with support for custom loss functions, logging, data loading, and more! A good introduction can be found in our two-part walkthrough (Part 1, Part 2).

from robustness import model_utils, datasets, train, defaults
from robustness.datasets import CIFAR

# We use cox (http://github.com/MadryLab/cox) to log, store and analyze
# results. Read more at https//cox.readthedocs.io.
from cox.utils import Parameters
import cox.store

# Hard-coded dataset, architecture, batch size, workers
ds = CIFAR('/path/to/cifar')
m, _ = model_utils.make_and_restore_model(arch='resnet50', dataset=ds)
train_loader, val_loader = ds.make_loaders(batch_size=128, workers=8)

# Create a cox store for logging
out_store = cox.store.Store(OUT_DIR)

# Hard-coded base parameters
train_kwargs = {
    'out_dir': "train_out",
    'adv_train': 1,
    'constraint': '2',
    'eps': 0.5,
    'attack_lr': 1.5,
    'attack_steps': 20
}
train_args = Parameters(train_kwargs)

# Fill whatever parameters are missing from the defaults
train_args = defaults.check_and_fill_args(train_args,
                        defaults.TRAINING_ARGS, CIFAR)
train_args = defaults.check_and_fill_args(train_args,
                        defaults.PGD_ARGS, CIFAR)

# Train a model
train.train_model(train_args, m, (train_loader, val_loader), store=out_store)

Note: robustness requires PyTorch to be installed with CUDA support.

Pretrained models

Along with the training code, we release a number of pretrained models for different datasets, norms and ε-train values. This list will be updated as we release more or improved models. Please cite this library (see bibtex entry below) if you use these models in your research.

For each (model, ε-test) combination we evaluate 20-step and 100-step PGD with a step size of 2.5 * ε-test / num_steps. Since these two accuracies are quite close to each other, we do not consider more steps of PGD. For each value of ε-test, we highlight the best robust accuracy achieved over different ε-train in bold.

Note #1: We did not perform any hyperparameter tuning and simply used the same hyperparameters as standard training. It is likely that exploring different training hyperparameters will increasse these robust accuracies by a few percent points.

Note #2: The pytorch checkpoint (.pt) files below were saved with the following versions of PyTorch and Dill:

torch==1.1.0
dill==0.2.9

CIFAR10 L2-norm (ResNet50):

CIFAR10 L2-robust accuracy

ε-train

ε-test

0.0

0.25

0.5

1.0

0.0

95.25% / -

92.77% / -

90.83% / -

81.62% / -

0.25

8.66% / 7.34%

81.21% / 81.19%

82.34% / 82.31%

75.53% / 75.53%

0.5

0.28% / 0.14%

62.30% / 62.13%

70.17% / 70.11%

68.63% / 68.61%

1.0

0.00% / 0.00%

21.18% / 20.66%

40.47% / 40.22%

52.72% / 52.61%

2.0

0.00% / 0.00%

0.58% / 0.46%

5.23% / 4.97%

18.59% / 18.05%

CIFAR10 Linf-norm (ResNet50):

CIFAR10 Linf-robust accuracy

ε-train

ε-test

0 / 255

8 / 255

0 / 255

95.25% / -

87.03% / -

8 / 255

0.00% / 0.00%

53.49% / 53.29%

16 / 255

0.00% / 0.00%

18.13% / 17.62%

ImageNet L2-norm (ResNet50):

  • ε = 0.0 (PyTorch pre-trained)

  • ε = 3.0

ImageNet L2-robust accuracy

ε-train

ε-test

0.0

3.0

0.0

76.13% / -

57.90% / -

0.5

3.35% / 2.98%

54.42% / 54.42%

1.0

0.44% / 0.37%

50.67% / 50.67%

2.0

0.16% / 0.14%

43.04% / 43.02%

3.0

0.13% / 0.12%

35.16% / 35.09%

ImageNet Linf-norm (ResNet50):

ImageNet Linf-robust accuracy

ε-train

ε-test

0.0

4 / 255

8 / 255

0 / 255

76.13% / -

62.42% / -

47.91% / -

4 / 255

0.04% / 0.03%

33.58% / 33.38%

33.06% / 33.03%

8 / 255

0.01% / 0.01%

13.13% / 12.73%

19.63% / 19.52%

16 / 255

0.01% / 0.01%

1.53% / 1.37%

5.00% / 4.82%

Citation

If you use this library in your research, cite it as follows:

@misc{robustness,
   title={Robustness (Python Library)},
   author={Logan Engstrom and Andrew Ilyas and Hadi Salman and Shibani Santurkar and Dimitris Tsipras},
   year={2019},
   url={https://github.com/MadryLab/robustness}
}

(Have you used the package and found it useful? Let us know!).

Maintainers

Contributors/Commiters

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

robustness-1.2.1.post2.tar.gz (81.7 kB view details)

Uploaded Source

Built Distribution

robustness-1.2.1.post2-py3-none-any.whl (95.3 kB view details)

Uploaded Python 3

File details

Details for the file robustness-1.2.1.post2.tar.gz.

File metadata

  • Download URL: robustness-1.2.1.post2.tar.gz
  • Upload date:
  • Size: 81.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.6.3

File hashes

Hashes for robustness-1.2.1.post2.tar.gz
Algorithm Hash digest
SHA256 425d8f2ba204aa7888396458fee013a814595991114d02a16e066863766444c6
MD5 f5f9fba0a020d5ff570c6cc166bb0de3
BLAKE2b-256 686d9b4186e7535d0d504e345801aca0563c7ef0b37c28d3e2bb242ce927b6fd

See more details on using hashes here.

File details

Details for the file robustness-1.2.1.post2-py3-none-any.whl.

File metadata

  • Download URL: robustness-1.2.1.post2-py3-none-any.whl
  • Upload date:
  • Size: 95.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.6.3

File hashes

Hashes for robustness-1.2.1.post2-py3-none-any.whl
Algorithm Hash digest
SHA256 a263950768a44f756e5d09bf32425c59d479e75ab036dff4f06298a7700c13b1
MD5 af22516d09b4da4af57adb1c608f5042
BLAKE2b-256 34d240eb05fa3f6ff7f84aa46b071f299dcc6ec7aa8ea93c57f2a110c3b1ca73

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page