robust-vision-benchmark 0.9.0

An HTTP server and client that provides access to Foolbox models and attacks.

Robust Vision Benchmark

This Python package provides utility functions to create submissions for the Robust Vision Benchmark and scripts to automatically test and upload them. You might also want to have a look at Foolbox, our python toolbox to benchmark the robustness of machine learning models using a large set of adversarial attacks.

Installation

We test using Python 2.7, 3.5 and 3.6. Other Python versions might work as well, but we recommend using Python 3.5 or newer.

pip install robust-vision-benchmark

Submitting a model

A model submission consists of a Dockerfile, a script (e.g. Python) and data (e.g. network weights).

Python script

Create a Python script that turns your model into a Foolbox model using one of our wrappers for TensorFlow, PyTorch, Theano, Keras, Lasagne, MXNet and starts the model_server.

from robust_vision_benchmark import model_server

# create your model
# ...

# turn it into a Foolbox model
model = foolbox.models.SomeModel(...)

# start the server
mnist_model_server(model)
cifar_model_server(model, channel_order='RGB or BGR')
imagenet_model_server(model, channel_order='RGB or BGR', image_size=224)

# For CIFAR and Imagenet, the channel_order must be set to either 'RGB' or 'BGR'.
# For ImageNet, the image_size must be set to an integer (usually 224 vor VGG-like networks and 299 for inception-like networks).

Dockerfile

Create a Dockerfile that installs all dependencies and starts the script.

FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04

RUN apt-get update ... && install ...

RUN pip3 install --no-cache-dir robust-vision-benchmark

...

COPY main.py main.py

CMD ["python3", "./main.py"]

For compatibility with our backend, please use a derivative of the nvidia/cuda:8.0 image (e.g. nvidia/cuda8.0-cudnn5-devel-ubuntu16.04) as the base image or contact us if you have special requirements.

Test the submission

Put the Dockerfile, script, data and other required files into a folder, e.g. model and run the following in your shell:

rvb-test-model model/

You can find an example in examples/model/.

Upload the submission

Once your model is ready for submission, upload it:

rvb-upload model/

Submitting

Go to https://robust.vision/benchmark/participate and put the URL returned by the upload script into the Submission URL.

Submitting an attack

An attack submission consists of a Dockerfile and a script (e.g. Python).

Python script

Create a Python script that implements your attack and starts the attack_server.

from robust_vision_benchmark import attack_server

# implement your attack
def attack(a):
    # ...

# start the server
attack_server(attack)

Dockerfile

Create a Dockerfile that installs all dependencies and starts the script.

FROM python:3.6

RUN pip3 install --no-cache-dir robust-vision-benchmark

...

COPY main.py main.py

CMD ["python3", "./main.py"]

Test the submission

Put the Dockerfile, script and other required files into a folder, e.g. attack and run the following in your shell:

rvb-test-attack attack/

You can find an example in examples/attack/.

Upload the submission

Once your attack is ready for submission, upload it:

rvb-upload attack/

Submitting

Go to https://robust.vision/benchmark/participate and put the URL returned by the upload script into the Submission URL.

Authors

• Jonas Rauber

• Wieland Brendel