gym_classics2

gym_classics2 provides classic finite Markov decision processes and readable reinforcement-learning algorithms for teaching with Gymnasium. It is based on Brett Daley's gym-classics and follows examples from Sutton and Barto's Reinforcement Learning: An Introduction. The package is intended for educational use. Example course material (slides, implementation examples and exercises) can be found here: Introduction Course to Reinforcement Learning.

The package includes:

• Gymnasium environments for gridworlds, random walks, mazes, and cliff walking.
• Direct access to the transition and reward model, $p(s',r\mid s,a)$.
• Textbook-oriented implementations of dynamic programming, Monte Carlo, temporal-difference, function-approximation, eligibility-trace, and policy-gradient algorithms.
• Gridworld plotting and animation helpers.

Installation

Python 3.10 or newer is required. Once published on PyPI, install the release with:

python -m pip install gym-classics2

To install the current development version:

python -m pip install "gym-classics2 @ git+https://github.com/mhahsler/gym-classics2.git"

Pygame rendering is optional. Install the render extra to use human or rgb_array rendering modes:

python -m pip install "gym-classics2[render]"

Quick start

Register the environments, create one with Gymnasium, and use the standard reset/step Gymnasium interface:

import gymnasium as gym
import numpy as np
import gym_classics2
gym_classics2.register()

seed = 42
rng = np.random.default_rng(seed)

env = gym.make("ClassicGridworld-v1", tabular=True)

state, info = env.reset(seed=seed)
for t in range(100):
    action = rng.integers(env.action_space.n)
    next_state, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated
    print(t, state, action, reward, next_state, done)
    state = next_state
    if done:
        state, info = env.reset()

env.close()

Model access

The unwrapped environments expose the full finite-MDP model. In tabular mode, states and next states are integer IDs:

gym_env = gym.make("ClassicGridworld-v1", tabular=True)
env = gym_env.unwrapped  # expose the gym_classics2-specific API

print("Starts:", env.start_states)
print("Goals:", env.goal_states)

state = env.state2id((0, 0))
action = env.action2id("up")
next_states, rewards, terminals, probabilities = env.model(state, action)

For this transition, the result is equivalent to:

next_states   = [0, 1, 3]
rewards       = [-0.04, -0.04, -0.04]
terminals     = [False, False, False]
probabilities = [0.1, 0.8, 0.1]

See the model-access guide for details.

Included gridworld environments

Included algorithms

All model-free algorithms can also be used with standard Gymnasium environments outside this package. See the algorithm guide for input requirements and return values.

Documentation and examples

• Documentation
• 4×3 Gridworld: value and policy iteration
• Frozen Lake: Monte Carlo methods
• Course materials

To build the documentation locally:

python -m pip install -e ".[docs]"
mkdocs serve

Maintainers: see the release instructions for distribution checks and PyPI publishing.

References

• Brett Daley (2021), Gym Classics.
• Michael Hahsler (2025), Introduction to Reinforcement Learning.
• Richard S. Sutton and Andrew G. Barto (2018), Reinforcement Learning: An Introduction, second edition.
• Mark Towers et al. (2024), Gymnasium: A Standard Interface for Reinforcement Learning Environments.