envpool 0.5.3

"C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments."

---

EnvPool is a C++-based batched environment pool with pybind11 and thread pool. It has high performance (~1M raw FPS in DGX on Atari games) and compatible APIs (supports both gym and dm_env, both sync and async, both single and multi player environment). Currently it supports:

• Atari games
• Mujoco
• Classic RL envs: CartPole, MountainCar, Pendulum, Acrobot
• Toy text RL envs: Catch, FrozenLake, Taxi, NChain, CliffWalking, Blackjack
• ViZDoom single player
• Box2D
• Procgen
• Minigrid

Here are EnvPool's several highlights:

• Compatible with OpenAI gym APIs and DeepMind dm_env APIs;
• Manage a pool of envs, interact with the envs in batched APIs by default;
• Support both synchronous execution and asynchronous execution;
• Support both single player and multi-player environment;
• Easy C++ developer API to add new envs;
• 1 Million Atari frames per second simulation with 256 CPU cores, ~13x throughput of Python subprocess-based vector env;
• ~3x throughput of Python subprocess-based vector env on low resource setup like 12 CPU cores;
• Comparing with existing GPU-based solution (Brax / Isaac-gym), EnvPool is a general solution for various kinds of speeding-up RL environment parallelization;
• Compatible with some existing RL libraries, e.g., Stable-Baselines3, Tianshou, or CleanRL.
  • Stable-Baselines3 Pendulum-v0 example;
  • Tianshou CartPole example and Pendulum-v0 example;
  • CleanRL Pong-v5 example (solving Pong in 5 mins (tracked experiment));
• Support customized C++ environment integration.

Installation

PyPI

EnvPool is currently hosted on PyPI. It requires Python >= 3.7.

You can simply install EnvPool with the following command:

$ pip install envpool

After installation, open a Python console and type

import envpool
print(envpool.__version__)

If no error occurs, you have successfully installed EnvPool.

From Source

Please refer to the guideline.

Documentation

The tutorials and API documentation are hosted on envpool.readthedocs.io.

The example scripts are under examples/ folder.

Benchmark Results

We perform our benchmarks with ALE Atari environment (with environment wrappers) on different hardware setups, including a TPUv3-8 virtual machine (VM) of 96 CPU cores and 2 NUMA nodes, and an NVIDIA DGX-A100 of 256 CPU cores with 8 NUMA nodes. Baselines include 1) naive Python for-loop; 2) the most popular RL environment parallelization execution by Python subprocess, e.g., gym.vector_env; 3) to our knowledge, the fastest RL environment executor Sample Factory before EnvPool.

We report EnvPool performance with sync mode, async mode, and NUMA + async mode, compared with the baselines on different number of workers (i.e., number of CPU cores). As we can see from the results, EnvPool achieves significant improvements over the baselines on all settings. On the high-end setup, EnvPool achieves 1 Million frames per second on 256 CPU cores, which is 13.3x of the gym.vector_env baseline. On a typical PC setup with 12 CPU cores, EnvPool's throughput is 2.8x of gym.vector_env.

Our benchmark script is in examples/benchmark.py. The detail configurations of 4 types of system are:

• Personal laptop: 12 core Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
• TPU-VM: 96 core Intel(R) Xeon(R) CPU @ 2.00GHz
• Apollo: 96 core AMD EPYC 7352 24-Core Processor
• DGX-A100: 256 core AMD EPYC 7742 64-Core Processor

Highest FPS	Laptop (12)	TPU-VM (96)	Apollo (96)	DGX-A100 (256)
For-loop	4,876	3,817	4,053	4,336
Subprocess	18,249	42,885	19,560	79,509
Sample Factory	27,035	192,074	262,963	639,389
EnvPool (sync)	40,791	175,938	159,191	470,170
EnvPool (async)	50,513	352,243	410,941	845,537
EnvPool (numa+async)	/	367,799	458,414	1,060,371

API Usage

The following content shows both synchronous and asynchronous API usage of EnvPool. You can also run the full script at examples/env_step.py

Synchronous API

import envpool
import numpy as np

# make gym env
env = envpool.make("Pong-v5", env_type="gym", num_envs=100)
# or use envpool.make_gym(...)
obs = env.reset()  # should be (100, 4, 84, 84)
act = np.zeros(100, dtype=int)
obs, rew, done, info = env.step(act)

Under the synchronous mode, envpool closely resembles openai-gym/dm-env. It has the reset and step functions with the same meaning. However, there is one exception in envpool: batch interaction is the default. Therefore, during the creation of the envpool, there is a num_envs argument that denotes how many envs you like to run in parallel.

env = envpool.make("Pong-v5", env_type="gym", num_envs=100)

The first dimension of action passed to the step function should equal num_envs.

act = np.zeros(100, dtype=int)

You don't need to manually reset one environment when any of done is true; instead, all envs in envpool have enabled auto-reset by default.

Asynchronous API

import envpool
import numpy as np

# make asynchronous
num_envs = 64
batch_size = 16
env = envpool.make("Pong-v5", env_type="gym", num_envs=num_envs, batch_size=batch_size)
action_num = env.action_space.n
env.async_reset()  # send the initial reset signal to all envs
while True:
    obs, rew, done, info = env.recv()
    env_id = info["env_id"]
    action = np.random.randint(action_num, size=batch_size)
    env.send(action, env_id)

In the asynchronous mode, the step function is split into two parts: the send/recv functions. send takes two arguments, a batch of action, and the corresponding env_id that each action should be sent to. Unlike step, send does not wait for the envs to execute and return the next state, it returns immediately after the actions are fed to the envs. (The reason why it is called async mode).

env.send(action, env_id)

To get the "next states", we need to call the recv function. However, recv does not guarantee that you will get back the "next states" of the envs you just called send on. Instead, whatever envs finishes execution gets recved first.

state = env.recv()

Besides num_envs, there is one more argument batch_size. While num_envs defines how many envs in total are managed by the envpool, batch_size specifies the number of envs involved each time we interact with envpool. e.g. There are 64 envs executing in the envpool, send and recv each time interacts with a batch of 16 envs.

envpool.make("Pong-v5", env_type="gym", num_envs=64, batch_size=16)

There are other configurable arguments with envpool.make; please check out envpool interface introduction.

Contributing

EnvPool is still under development. More environments will be added, and we always welcome contributions to help EnvPool better. If you would like to contribute, please check out our contribution guideline.

License

EnvPool is under Apache2 license.

Other third-party source-code and data are under their corresponding licenses.

We do not include their source code and data in this repo.

Citing EnvPool

If you find EnvPool useful, please cite it in your publications.

@misc{envpool,
  author = {Jiayi Weng and Min Lin and Zhongwen Xu and Shuicheng Yan},
  title = {EnvPool},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/sail-sg/envpool}},
}

Disclaimer

This is not an official Sea Limited or Garena Online Private Limited product.