POPGym Arcade

POPGym Arcade is a GPU-accelerated Atari-style benchmark and suite of analysis tools for reinforcement learning.

For more details, check out the project website.

Check the documentation for the guide you need — quick start, memory introspection, or reproducing experiments.

Tasks

POPGym Arcade contains pixel-based tasks in the style of the Arcade Learning Environment.

Each environment provides:

• Three difficulty settings
• One observation and action space shared across all envs
• Fully observable and partially observable configurations
• Fast and easy GPU vectorization using jax
• Standardized returns in [0,1] or [-1, 1]

Baselines

We provide a single training script for all algorithms and memory models. The memax library provides 18 different memory models for use in our script.

RL Algorithms

• PQN
• PPO
• DQN

Getting Started

To install the environments, run

pip install popgym-arcade

If you plan to use our training scripts, install the baselines as well. If you want to play the games yourself, also use the human flag.

pip install 'popgym-arcade[baselines,human]'

[!NOTE] If you do not already have jax installed, we install CPU jax by default. For GPU acceleration, run pip install jax[cuda12] after installing popgym-arcade.

Human Play

The play script installed with pip install popgym-arcade[human] lets you play the games yourself using the arrow keys and spacebar.

popgym-arcade-play NoisyCartPoleEasy        # play MDP 256 pixel version
popgym-arcade-play BattleShipEasy -p -o 128 # play POMDP 128 pixel version

Other Useful Libraries

• stable-gymnax - A (stable) jax-capable gymnasium API
• memax - Recurrent models for jax
• popgym - The original collection of POMDPs, implemented in numpy
• popjaxrl - A jax version of popgym
• popjym - A more readable version of popjaxrl environments that served as a basis for our work

Citation

If you use POPGym Arcade in your work, please cite it as follows:

@article{wang2025investigating,
  title={Investigating Memory in Model-Free RL with POPGym Arcade},
  author={Wang, Zekang and He, Zhe and Zhang, Borong and Toledo, Edan and Morad, Steven},
  journal={arXiv preprint arXiv:2503.01450},
  year={2025}
}