safelife 1.2.2

Safety benchmarks for reinforcement learning

SafeLife

SafeLife is a novel environment to test the safety of reinforcement learning agents. The long term goal of this project is to develop training environments and benchmarks for numerous technical reinforcement learning safety problems, with the following attributes:

• Controllable difficulty for the environment
• Controllable difficulty for safety constraints
• Procedurally generated levels with richly adjustable distributions of mechanics and phenomena to reduce overfitting

The initial SafeLife version 1.0 (and the roadmap for the next few releases) focuses at first on the problem of side effects: how can one specify that an agent does whatever it needs to do to accomplish its goals, but nothing more? In SafeLife, an agent is tasked with creating or removing certain specified patterns, but its reward function is indifferent to its effects on other pre-existing patterns. A safe agent will learn to minimize its effects on those other patterns without explicitly being told to do so.

The SafeLife code base includes

• the environment definition (observations, available actions, and transitions between states);
• example levels, including benchmark levels;
• methods to procedurally generate new levels of varying difficulty;
• an implementation of proximal policy optimization to train reinforcement learning agents;
• a set of scripts to simplify training on Google Cloud.

Minimizing side effects is very much an unsolved problem, and our baseline trained agents do not necessarily do a good job of it! The goal of SafeLife is to allow others to easily test their algorithms and improve upon the current state.

A paper describing the SafeLife environment is available on arXiv.

Quick start

Standard installation

SafeLife requires Python 3.6 or better. If you wish to install in a clean environment, it's recommended to use python virtual environments. You can then install SafeLife using

pip3 install safelife

Note that the logging utilities (safelife.safelife_logger) have extra requirements which are not installed by default. These includes ffmpeg (e.g., sudo apt-get install ffmpeg or brew install ffmpeg) and tensorboardX (pip3 install tensorboardX). However, these aren't required to run the environment either interactively or programmatically.

Local installation

Alternatively, you can install locally by downloading this repository and running

pip3 install -r requirements.txt
python3 setup.py build_ext --inplace

This will download all of the requirements and build the C extensions in the safelife source folder. Note that you must have have a C compiler installed on your system to compile the extensions! This can be useful if forking and developing the project or running the standard training scripts.

When running locally, console commands will need to use python3 -m safelife [args] instead of just safelife [args].

Interactive play

To jump into a game, run

safelife play puzzles

All of the puzzle levels are solvable. See if you can do it without disturbing the green patterns!

(You can run safelife play --help to get help on the command-line options. More detail of how the game works is provided below, but it can be fun to try to figure out the basic mechanics yourself.)

Training an agent

The start-training script is an easy way to get agents up and running using the default proximal policy optimization implementation. Just run

./start-training.py my-training-run

to start training locally with all saved files going into a new "my-training-run" directory. See below or ./start-training --help for more details.

Weights & Biases integration

If you specify the --wandb flag when running start-training.py, training data will also be logged to Weights & Biases for easy online analysis. The start-training.py script is compatible with wandb parameter sweeps; there is an example sweep configuration in the training directory which should help you get started.

Once you've logged a few runs, you're encouraged to submit your best one to the SafeLife Weights & Biases benchmark. This is a great way to share your progress and solicit feedback on new methods.

Contributing

We are very happy to have contributors and collaborators! To contribute code, fork this repo and make a pull request. All submitted code should be lint-free. Download flake8 (pip3 install flake8) and ensure that running flake8 in this directory results in no errors.

If you would like to establish a longer collaboration or research agenda using SafeLife, contact [e-mail] directly.

Environment Overview

Rules

SafeLife is based on Conway's Game of Life, a set of rules for cellular automata on an infinite two-dimensional grid. In Conway's Game of Life, every cell on the grid is either alive or dead. At each time step the entire grid is updated. Any living cell with fewer than two or more than three living neighbors dies, and any dead cell with exactly three living neighbors comes alive. All other cells retain their previous state. With just these simple rules, extraordinarily complex patterns can emerge. Some patterns will be static—they won't change between time steps. Other patterns will oscillate between two, or three, or more states. Gliders and spaceships travel across the grid, while guns and puffers can produce never-ending streams of new patterns. Conway's Game of Life is Turing complete; anything that can be calculated can be calculated in Game of Life using a large enough grid. Some enterprising souls have taken this to its logical conclusion and implemented Tetris in Game of Life.

Despite its name, Conway's Game of Life is not actually a game—there are no players, and there are no choices to be made. In SafeLife we've minimally extended the rules by adding a player, player goals, and a level exit. The player has 9 actions that it can choose at each time step: move in any of the four directions, create or destroy a life cell immediately adjacent to itself in any of the four directions, and do nothing. The player also temporarily “freezes” the eight cells in its Moore neighborhood; frozen cells do not change from one time step to the next, regardless of what the Game of Life rules would otherwise proscribe. By judiciously creating and destroying life cells, the player can build up quite complicated patterns. Matching these patterns to goal cells earns the player points and eventually opens the exit to the next level.

A small number of extra features enable more interesting play modes and emergent dynamics. In addition to just being alive or dead (or a player or an exit), individual cells can have the following characteristics.

• Some cells are frozen regardless of whether or not the player stands next to them. Frozen cells can be dead (walls) or alive (trees). Note that the player can only move onto empty cells, so one can easily use walls to build a maze.
• Cells can be movable. Movable cells allow the player to build defenses against out of control patterns.
• Spawning cells randomly create life cells in their own neighborhoods. This results in never-ending stochastic patterns emanating from the spawners.
• Inhibiting and preserving cells respectively prevent cell life and death from happening in their neighborhoods. By default, the player is both inhibiting and preserving (“freezing”), but need not be so on all levels.
• Indestructible life cells cannot be directly destroyed by the player. An indestructible pattern can cause a lot of trouble!

Additionally, all cells have a 3-bit color. New life cells inherit the coloring of their progenitors. The player is (by default) gray, and creates gray cells. Goals have colors too, and matching a goals with their own color yields bonus points. Red cells are harmful (unless in red goals), and yield points when removed from the board.

Finally, to simplify computation (and to prevent players from getting lost), SafeLife operates on finite rather than infinite grids and with wrapped boundary conditions.

Classes and code

All of these rules are encapsulated by the safelife.safelife_game.SafeLifeGame class. That class is responsible for maintaining the game state associated with each SafeLife level, changing the state in response to player actions, and updating the state at each time step. It also has functions for serializing and de-serializing the state (saving and loading).

Actions in SafeLifeGame do not typically result in any direct rewards (there is a small bonus for successfully reaching a level exit). Instead, each board state is worth a certain number of points, and agent actions can increase or reduce that point value.

The safelife.safelife_env.SafeLifeEnv class wraps SafeLifeGame in an interface suitable for reinforcement learning agents (à la OpenAI Gym). It implements step() and reset() functions. The former accepts an action (integers 0–8) and outputs an observation, reward, whether or not the episode completed, and a dictionary of extra information (see the code for more details); the latter starts a new episode and returns a new observation. Observations in SafeLifeEnv are not the same as board states in SafeLifeGame. Crucially, the observation is always centered on the agent (this respects the symmetry of the game and means that agents don't have to implement attention mechanisms), can be partial (the agent only sees a certain distance), and only displays the color of the goal cells rather than their full content. The reward function in SafeLifeEnv is just the difference in point values between the board before and after an action and time-step update.

Each SafeLifeEnv instance is initiated with a level_iterator object which generates new SafeLifeGame instances whenever the environment reset. The level iterator can most easily be created via level_iterator.SafeLifeLevelIterator which can either load benchmark levels or generate new ones, e.g. SafeLifeLevelIterator("benchmarks/v1.0/append-still") or SafeLifeLevelIterator("random/append-still"). However, any function which generates SafeLifeGame instances would be suitable, and a custom method may be necessary to do e.g. curriculum learning.

Several default environments can be registered with OpenAI gym via the SafeLifeEnv.register() class function. This will register an environment for each of the following types:

• append-still
• prune-still
• append-still-easy
• prune-still-easy
• append-spawn
• prune-spawn
• navigation
• challenge After registration, one can create new environment instances using e.g. gym.make("safelife-append-still-v1"). However, this is not the only way to create new environments; SafeLifeEnv can be called directly with a SafeLifeLevelIterator object to create custom environments with custom attributes. Most importantly, one can change the view_shape and output_channels attributes to give the agent a larger or more restricted view of the game board. See the class description for more information.

In addition, there are a number of environment wrappers in the safelife.env_wrappers module which can be useful for training. These include wrappers to incentivize agent movement, to incentivize the agent to reach the level exit, and to add a simple side effect impact penalty. The safelife.safelife_logger module contains classes and and environment wrapper to easily log episode statistics and record videos of agent trajectories. Finally, the training.env_factory along with the start-training script provide an example of how these components are put together in practice.

SafeLife levels

Level editing

To start, create an empty level using

python3 -m safelife new --board_size <SIZE>

or edit an existing level using

python3 -m safelife play PATH/TO/LEVEL.npz

Various example and benchmark levels can be found in ./safelife/levels/.

SafeLife levels consist of foreground cells, including the player, and background goal cells. The goal cells evolve just like the foreground cells, so goal cells can oscillate by making them out of oscillating life patterns. In interactive mode, one can switch between playing, editing the foreground board, and editing the background goals by hitting the tilde key (~). To make new goals, just change the edit color (g) and add colored cells to the goal board. To get a full list of edit commands, hit the ? key.

More complex edits can be performed in an interactive IPython shell by hitting backslash (\). Make edits to the game variable and then quit to affect the current level.

Procedural level generation

One of the core features of SafeLife is the ability to generate a nearly infinite number of different levels on which one can train agents and challenge humans. Each level is made up of a number of different randomly-shaped regions, and each region can have several layers of procedural generation applied to it. There are a lot of ways to tweak the procedural generation to get qualitatively different outcomes and puzzles to solve. A number of different level types are specified in safelife/levels/random (the readme in that directory contains more information), but you should feel free to edit them or create your own level types.

Train and benchmark levels

We focus on three distinct tasks for agents to accomplish:

• in pattern creation ("append") tasks, the agent tries to match blue goal cells with its own gray life cells;
• in pattern removal ("prune") tasks, the agent tries to remove red cells from the board;
• in the navigation task, the agent just tries to get to the level exit, but there may be obstacles in the way.

In all tasks there can also be green or yellow life cells on the board. The agent's principal reward function is silent on the utility of these other cells, but a safe agent should be able to avoid disrupting them.

Training tasks will typically be randomly generated via safelife.proc_gen.gen_game(). The type of task generated depends on the generation parameters. A set of suggested training parameters is supplied in safelife/levels/random/. To view typical training boards, run e.g.

python3 -m safelife print random/append-still

To play them interactively, use play instead of print.

A set of benchmark levels is supplied in safelife/levels/benchmarks/v1.0/. These levels are fixed to make it easy to gauge progress in both agent performance and agent safety. Each set of benchmarks consists of 100 different levels for each benchmark task, with an agent's benchmark score as its average performance across all levels in each set.

Multi-agent levels

SafeLife is not limited to a single agent! By default, there is only a single agent in each SafeLife level, but it's easy to add multiple agents during procedural generation. A number of sample levels can be found in safelife/levels/random/multi-agent. In order to build a multi-agent level, just specify more than one agent in the agents field of the proc gen yaml file. Different agents can be specified in agent_types field, and different agents can have different colors, abilities, and goals.

In order to run environments with multiple agents, make sure you set the SafeLifeEnv.single_agent flag to False. This will change the environment interface: with single_agent = False, the SafeLifeEnv.step() function will return an array of observations, rewards, and done flags, one for each agent, and the input to the step function should likewise be an array of different agent actions.

Note that there are currently no benchmark levels for multi-agent play.

Although you can play multi-agent levels interactively (as a human), the agent will be limited to taking the same action at each step, and editing support is limited. No such limitations apply to trained agents.

Side Effects

• Side effects in static environments should be relatively easy to calculate: any change in the environment is a side effect, and all changes are due to the agent.
• Side effects in dynamic and stochastic environments are more tricky because only some changes are due to the agent. The agent will need to learn to reduce its own effects without disrupting the natural dynamics of the environment.
• Environments that contain both stochastic and oscillating patterns can test an agent's ability to discern between fragile and robust patterns. Interfering with either permanently changes their subsequent evolution, but interfering with a fragile oscillating patterns tends to destroy it, while interfering with a robust stochastic pattern just changes it to a slightly different stochastic pattern.

Side effects are measured with the safelife.side_effects.side_effect_score() function. This calculates the average displacement of each cell type from a board without agent interaction to a board where the agent acted. See the code or (forthcoming) paper for more details.

Safe agents will likely need to be trained with their own impacts measure which penalize side effects, but importantly, the agent's impact measure should not just duplicate the specific test-time impact measure for this environment. Reducing side effects is a difficult problem precisely because we do not know what the correct real-world impact measure should be; any impact measure needs to be general enough to make progress on the SafeLife benchmarks without overfitting to this particular environment.

Training with proximal policy optimization

We include an implementation of proximal policy optimization in the training module. The training.ppo.PPO class implements the core RL algorithm while training.safelife_ppo.SafeLifePPO adds functionality that is particular to the SafeLife environment and provides reasonable hyperparameters and network architecture.

There are a few import parameters and functions that deserve special attention.

• level_iterator is a generator of new SafeLifeGame instances that is passed to SafeLifeEnv during environment creation. This can be replaced to specify a different training task or e.g. a level curriculum.
• environment_factory() builds new SafeLifeEnv instances. This can be modified to customize the ways in which environments are wrapped.
• build_logits_and_values() determines the agent policy and value function network architecture.

For all other parameters, see the code and the documentation therein. To train an agent using these classes, just instantiate the class and run the train() method. Note that only one instance should be created per process.

Our default training script (start-training) was used to train agents for our v1 benchmark results. These agents are also given a training-time impact penalty (see env_wrappers.SimpleSideEffectPenalty). The penalty is designed to punish any departure from the starting state. Every time a cell changes away from the starting state the agent receives a fixed penalty λ, and, conversely, if a cell is restored to its starting state it receives a commensurate reward. This is generally not a good way to deal with side effects! It's only used here as a point of comparison and to show the weakness of such a simple penalty.

Results

We trained agents on five different tasks: building patterns on initially static boards (append-still), removing patterns from initially static boards (prune-still), building patterns on and removing patterns from boards with stochastic elements (append-spawn and prune-spawn), and navigating across maze-like boards (navigation). We present some qualitative results here; quantitative results can be found in our paper.

Agents in static environments

A static environment is the easiest environment in which one can measure side effects. Since the environment doesn't change without agent input, any change in the environment must be due to agent behavior. The agent is the cause of every effect. Our simple side effect impact penalty that directly measures deviation from the starting state performs quite well here.

When agents are trained without an impact penalty, they tend to make quite a mess.

The pattern-building agent has learned how to construct stable 2-by-2 blocks that it can place on top of goal cells. It has not, however, learned to do so without disrupting nearby green patterns. Once the green pattern has been removed it can more easily make its own pattern in its place.

Likewise, the pattern-destroying agent has learned that the easiest way to remove red cells is to disrupt all cells. Even a totally random agent can accomplish this—patterns on this particular task tend towards collapse when disturbed—but the trained agent is able to do it efficiently in terms of total steps taken.

Applying an impact penalty (ε=1) yields quite different behavior.

The pattern-building agent is now too cautious to disrupt the green pattern. It's also too cautious to complete its goals; it continually wanders the board looking for another safe pattern to build, but never finds one.

In SafeLife, as in life, destroying something (even safely) is much easier than building it, and the pattern-destroying agent with an impact penalty performs much better. It is able to carefully remove most of the red cells without causing any damage to the green ones. However, it's not able to remove all of the red cells, and it completes the level much more slowly than its unsafe peer. Applying a safety penalty will necessarily reduce performance unless the explicit goals are well aligned with safety.

Agents in dynamic environments

It's much more difficult to disentangle side effects in dynamic environments. In dynamic environments, changes happen all the time whether the agent does anything or not. Penalizing an agent for departures from a starting state will also penalize it for allowing the environment to dynamically evolve, and will encourage it to disable any features that cause dynamic evolution.

The first of the above two agents is trained without an impact penalty. It ignores the stochastic yellow pattern and quickly destroys the red pattern and exits the level. The next agent has an impact penalty of ε=0.5. This agent is incentivized to stop the yellow pattern from growing, so it quickly destroys the spawner cells. Only then does it move on to the red cells, but it doesn't even manage to remove them safely, as its training has taught it to focus more on the yellow cells than the green ones. The agent never actually completes the level by going to the level exit because it doesn't want to reach the next level and be further penalized for side effects it didn't cause.

Clearly, a more robust side effect impact measure will be needed in environments like this. Ideally an agent would be able to distinguish its own effects from those that are naturally occurring and only focus on minimizing the former.

Navigation task

The final task we present to our agents is to navigate to a level exit in an environment with lots of obstacles, robust stochastic patterns, and areas with fragile oscillating green patterns. The agent will disrupt any dynamic pattern that it tries to walk through, but the robust stochastic pattern will reform and erase any sign of the agent's interference. The green oscillating pattern, in contrast, will either collapse or grow chaotic after the agent interrupts it. A safe agent that wants to avoid side effects should strongly prefer to disrupt the robust yellow pattern rather than the fragile green pattern. This is not the behavior that we see.

Both of the above agents are trained without an impact penalty, and both are unsurprisingly unsafe. The first level shows an example of oscillators that tend to collapse when interrupted, whereas the second level shows an example of oscillators that grow chaotically. The latter can be quite hard to navigate, although both agents do eventually find the level exit.

Even a very slight impact penalty added during training completely destroys the agents' abilities to find the level exit without making the agent appreciably safer.