popgym 1.0.2

A collection of partially-observable procedural gym environments

POPGym: Partially Observable Process Gym

POPGym is designed to benchmark memory in deep reinforcement learning. It contains a set of environments and a collection of memory model baselines. The full paper is available on OpenReview. Please see the documentation for advanced installation instructions and examples.

Quickstart Install

pip install popgym # base environments only, only requires numpy and gymnasium
pip install "popgym[navigation]" # also include navigation environments, which require mazelib
pip install "popgym[baselines]" # environments and memory baselines

POPGym Environments

POPGym contains Partially Observable Markov Decision Process (POMDP) environments following the Openai Gym interface. POPGym environments have minimal dependencies and fast enough to solve on a laptop CPU in less than a day. We provide the following environments:

Environment	Tags	Temporal Ordering	Colab FPS	Macbook Air (2020) FPS
Battleship (Code)	Game	None	117,158	235,402
Concentration (Code)	Game	Weak	47,515	157,217
Higher Lower (Code)	Game, Noisy	None	24,312	76,903
Labyrinth Escape (Code)	Navigation	Strong	1,399	41,122
Labyrinth Explore (Code)	Navigation	Strong	1,374	30,611
Minesweeper (Code)	Game	None	8,434	32,003
Multiarmed Bandit (Code)	Noisy	None	48,751	469,325
Autoencode (Code)	Diagnostic	Strong	121,756	251,997
Count Recall (Code)	Diagnostic, Noisy	None	16,799	50,311
Repeat First (Code)	Diagnostic	None	23,895	155,201
Repeat Previous (Code)	Diagnostic	Strong	50,349	136,392
Stateless Cartpole (Code)	Control	Strong	73,622	218,446
Noisy Stateless Cartpole (Code)	Control, Noisy	Strong	6,269	66,891
Stateless Pendulum (Code)	Control	Strong	8,168	26,358
Noisy Stateless Pendulum (Code)	Control, Noisy	Strong	6,808	20,090

Feel free to rerun this benchmark using this colab notebook.

POPGym Baselines

POPGym baselines implements recurrent and memory model in an efficient manner. POPGym baselines is implemented on top of rllib using their custom model API. We provide the following baselines:

1. MLP
2. Positional MLP
3. Framestacking (Paper)
4. Temporal Convolution Networks (Paper)
5. Elman Networks (Paper)
6. Long Short-Term Memory (Paper)
7. Gated Recurrent Units (Paper)
8. Independently Recurrent Neural Networks (Paper)
9. Fast Autoregressive Transformers (Paper)
10. Fast Weight Programmers (Paper)
11. Legendre Memory Units (Paper)
12. Diagonal State Space Models (Paper)
13. Differentiable Neural Computers (Paper)

Leaderboard

The leaderboard is available at paperswithcode.

Contributing

Follow style and ensure tests pass

pip install pre-commit
pre-commit install
pytest popgym/tests

Citing

@inproceedings{
morad2023popgym,
title={{POPG}ym: Benchmarking Partially Observable Reinforcement Learning},
author={Steven Morad and Ryan Kortvelesy and Matteo Bettini and Stephan Liwicki and Amanda Prorok},
booktitle={The Eleventh International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=chDrutUTs0K}
}