RoAd-RL: Robust Adversarial Deep Reinforcement Learning

A Python library for benchmarking adversarial attacks and defenses in deep reinforcement learning. RoAd-RL provides a modular, reproducible pipeline for training RL agents, evaluating them under observation-space adversarial attacks, and applying inference-time defenses.

Installation

pip install road-rl

Install optional extras based on what you need:

pip install "road-rl[torch]"       # PyTorch support
pip install "road-rl[sb3]"         # Stable-Baselines3
pip install "road-rl[rl]"          # Gymnasium
pip install "road-rl[box2d]"       # Box2D environments (LunarLander etc.)
pip install "road-rl[plots]"       # scienceplots for publication figures
pip install "road-rl[config]"      # YAML config support

Or install everything at once:

pip install -r requirements.txt

Library Overview

RoAd-RL evaluates robustness by running episodes under a sweep of attack budgets (epsilon values) and seeds. At each step:

observation → [attack] → [defense] → policy → action

Attacks (observation-space, gradient-based):

Defenses (inference-time preprocessing):

Supported environments:

Repository Structure

road-rl/
├── road_rl/                  # Library package
│   ├── attacks/              # FGSM, PGD, JSMA adversarial attacks
│   ├── defenses/             # Normalize/clip, smoothing, gaussian noise, feature squeeze, PCA
│   ├── policies/             # Policy adapters: DQN, PPO, SAC, SB3
│   ├── envs/                 # Environment adapters: Gym, Highway, Atari
│   ├── eval/                 # Episode runner, sweep runner, evaluator
│   ├── metrics/              # Robustness, return, risk, safety metrics
│   ├── adv_training/         # Adversarial training wrappers and schedulers
│   ├── train/                # DQN/PPO/SAC trainers via Stable-Baselines3
│   ├── io/                   # Config loader, CSV/JSON logger, plotting
│   ├── utils/                # Training curve aggregation and plotting
│   └── cli/                  # `road-rl eval` CLI entry point
│
├── scripts/
│   ├── train_all.py                  # Batch training
│   ├── eval_all.py                   # Batch evaluation
│   ├── run_eval.py                   # Sequential or tmux sweep runner
│   ├── eval_sweep.py                 # Single sweep evaluation
│   ├── eval_rewards_only.py          # Clean reward evaluation helper
│   ├── eval_highway_vanilla.py       # Highway vanilla combined eval
│   ├── eval_highway_attacks.py       # Highway under attacks combined eval
│   ├── plot_all.py                   # Aggregate plots from episode CSVs
│   ├── plot_training_benchmarks.py   # Training curve benchmark plots
│   ├── highway_{dqn,ppo,sac}_main.py # Highway training entrypoints
│   ├── lunar_{dqn,ppo,sac}_main.py   # Lunar training entrypoints
│   └── release_check.py              # Pre-release validation
│
└── tests/                    # Unit tests

Quick Start

Programmatic sweep

from road_rl.eval.sweep_runner import run_sweep
from road_rl.envs.make_env import EnvSpec, make_env_factory
from road_rl.train.utils import build_attack, build_defense, load_policy_from_checkpoint

env_factory = make_env_factory(EnvSpec("LunarLander-v2", adapter="gym"))
policy = load_policy_from_checkpoint("checkpoints/lunar_dqn.zip", algorithm="dqn")
attack = build_attack("fgsm", norm="linf")
defense = build_defense("normalize_clip")

result = run_sweep(
    env_factory=env_factory,
    policy=policy,
    env_id="LunarLander-v2",
    algorithm="dqn",
    epsilons=[0.0, 0.01, 0.05],
    seeds=[0, 1, 2],
    attack=attack,
    defense=defense,
    episodes_per_seed=30,
    show_progress=True,
)
# result.episodes → list of EpisodeResult

CLI evaluation

road-rl eval \
  --env-id LunarLander-v2 \
  --adapter gym \
  --policy-path checkpoints/lunar_dqn.zip \
  --algorithm dqn \
  --eps 0.0 0.005 0.01 0.025 0.05 \
  --seeds 0 1 2 \
  --episodes-per-seed 40 \
  --attack fgsm \
  --defense none \
  --out results/

Training agents

# Train a specific environment
python scripts/lunar_dqn_main.py
python scripts/highway_ppo_main.py

# Batch train from configs
python scripts/train_all.py

Extending the Library

Custom policy:

from road_rl.policies.base import Policy
import numpy as np

class MyPolicy(Policy):
    def act(self, obs: np.ndarray) -> int:
        ...
    def loss(self, obs_tensor) -> "torch.Tensor":
        ...  # differentiable loss for gradient-based attacks

Custom attack:

from road_rl.attacks.base import Attack
from road_rl.core.context import StepContext

class MyAttack(Attack):
    def apply(self, obs, policy, ctx: StepContext):
        ...

Custom defense:

from road_rl.defenses.base import Defense

class MyDefense(Defense):
    def apply(self, obs, ctx):
        ...

License

MIT — see LICENSE.