HASARD 0.1.0

Egocentric 3D Safe Reinforcement Learning Benchmark

HASARD: A Benchmark for Harnessing Safe Reinforcement Learning with Doom

HASARD (Harnessing Safe Reinforcement Learning with Doom) is a benchmark for Safe Reinforcement Learning within complex, egocentric perception 3D environments derived from the classic DOOM video game. It features 6 diverse scenarios each spanning across 3 levels of difficulty. A short demo of HASARD is available on Youtube.

Scenario	Level 1	Level 2	Level 3
Armament Burden
Detonator’s Dilemma
Volcanic Venture
Precipice Plunge
Collateral Damage
Remedy Rush

Key Features

• Egocentric Perception: Agents learn solely from first-person pixel observations under partial observability.
• Beyond Simple Navigation: Whereas prior benchmarks merely require the agent to reach goal locations on flat surfaces while avoiding obstacles, HASARD necessitates comprehending complex environment dynamics, anticipating the movement of entities, and grasping spatial relationships.
• Dynamic Environments: HASARD features random spawns, unpredictably moving units, and terrain that is constantly moving or periodically changing.
• Difficulty Levels: Higher levels go beyond parameter adjustments, introducing entirely new elements and mechanics.
• Reward-Cost Trade-offs: Rewards and costs are closely intertwined, with tightening cost budget necessitating a sacrifice of rewards.
• Safety Constraints: Each scenario features a hard constraint setting, where any error results in immediate in-game penalties.
• Focus on Safety: Achieving high rewards is straightforward, but doing so while staying within the safety budget demands learning complex and nuanced behaviors.

Policy Visualization

HASARD enables overlaying a heatmap of the agent's most frequently visited locations providing further insights into its policy and behavior within the environment. These examples show how an agent navigates Volcanic Venture, Remedy Rush, and Armament Burden during the course of training:

Augmented Observations

HASARD supports augmented observation modes for further visual analysis. By utilizing privileged game state information, it can generate simplified observation representations, such as segmenting objects in the scene or rendering the environment displaying only depth from surroundings.

Installation

To install HASARD, simply clone or download the repository and run:

$ pip install .

Getting Started

Below we provide a short code snippet to run a HASARD task.

import hasard

env = hasard.make('ArmamentBurdenLevel1-v0')
env.reset()
terminated = truncated = False
steps = total_cost = total_reward = 0
while not (terminated or truncated):
    action = env.action_space.sample()
    state, reward, cost, terminated, truncated, info = env.step(action)
    env.render()
    steps += 1
    total_cost += cost
    total_reward += reward
print(f"Episode finished in {steps} steps. Reward: {total_reward:.2f}. Cost: {total_cost:.2f}")
env.close()

Acknowledgements

HASARD environments are built on top of the ViZDoom platform.
Our Safe RL baseline methods are implemented in Sample-Factory.
Our experiments were managed using WandB.