Skip to main content

Adversarial Robustness Test Bed

Project description


GitHub CI PyPI Status Badge PyPI - Python Version License: MIT Code style: black

Overview

ARMORY is a test bed for running scalable evaluations of adversarial defenses. Configuration files are used to launch local or cloud instances of the ARMORY docker containers. Models, datasets, and evaluation scripts can be pulled from external repositories or from the baselines within this project.

Our evaluations are created so that attacks and defenses may be interchanged. To do this we standardize all attacks and defenses as subclasses of their respective implementations in the adversarial-robustness-toolbox

Installation & Configuration

pip install armory-testbed

Upon installing armory, a directory will be created at ~/.armory. This user specific folder is the default directory for downloaded datasets, model weights, and evaluation outputs.

To change these default directories simply run armory configure after installation.

Usage

There are four ways to interact with the armory container system.

  1. armory run <path/to/config.json>. This will run a configuration file end to end. Stdout and stderror logs will be displayed to the user, and the container will be removed gracefully upon completion. Results from the evaluation can be found in your output directory.

  2. armory launch <tf1|tf2|pytorch> --interactive. This will launch a framework specific container, with appropriate mounted volumes, for the user to attach to for debugging purposes. A command to attach to the container will be returned from this call, and it can be ran in a separate terminal. To later close the interactive container simply run CTRL+C from the terminal where this command was ran.

  3. armory launch <tf1|tf2|pytorch> --jupyter. Similar to the interactive launch, this will spin up a container for a specific framework, but will instead return the web address of a jupyter lab server where debugging can be performed. To close the jupyter server simply run CTRL+C from the terminal where this command was ran.

  4. armory exec <tf1|tf2|pytorch> -- <cmd>. This will run a specific command within a framework specific container. A notable use case for this would be to run test cases using pytest. After completion of the command the container will be removed.

Note: Since ARMORY launches Docker containers, the python package must be ran on system host (i.e. not inside of a docker container).

Example usage:

pip install armory-testbed
armory configure
git clone https://github.com/twosixlabs/armory-example.git
cd armory-example
git checkout tags/v0.5.0 -b v0.5
armory run examples/fgm_attack.json

What is available in the container:

All containers have a pre-installed armory package installed so that baseline models, datasets, and scenarios can be utilized.

Additionally, volumes (such as your current working directory) will be mounted from your system host so that you can modify code to be ran, and retrieve outputs. For more information on these mounts, please see our Docker documentation

Scenarios

Armory provides several baseline threat-model scenarios for various data modalities. When running an armory configuration file, the robustness of a defense will be evaluated against that given scenario. For more information please see our Scenario Documentation.

FAQs

Please see the frequently asked questions documentation for more information on:

  • Datasets returning NumPy arrays
  • Access to underlying models from wrapped classifiers.

Contributing

Armory is an open source project and as such we welcome contributions! Please refer to our contribution docs for how to get started.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for armory-testbed, version 0.6.0rc0
Filename, size File type Python version Upload date Hashes
Filename, size armory_testbed-0.6.0rc0-py3-none-any.whl (87.7 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page