deep-rl-box 0.0.1

a deep reinforcement learning framework

Deep RL Box

A collection of Deep Reinforcement Learning algorithms implemented with Tensorflow to solve Atari games and classic control tasks like CartPole, LunarLander, and MountainCar.

The overall project structure was based on DeepMind's DQN Zoo. We adapted the code to support Tensorflow, in addition also implemented some SOTA algorithms like PPO, RND, R2D2, and Agent57.

Content

• Environment and Requirements
• Implemented Algorithms
• Code Structure
• Author's Notes
• Quick Start
• Train Agents
• Evaluate Agents
• Monitoring with Tensorboard
• Acknowledgments
• License
• Citing our work

Environment and Requirements

• Python 3.10.6
• pip 23.0.1
• Tensorflow 2.10.0
• gymnasium 1.1.1
• Tensorboard 2.13.0

Implemented Algorithms

Step 1: Fundamentals Actor-Critic (1983) – Understand policy gradient methods and value-based learning.

REINFORCE (1987) – Learn about Monte Carlo policy gradients.

REINFORCE-Baseline (1992) – Grasp variance reduction techniques.

Step 2: Deep Q-Learning DQN (2013) – Get familiar with deep reinforcement learning and experience replay.

Double-DQN, Prioritized-DQN (2015) – Explore stability improvements in Q-learning.

DRQN (2015) – Learn about recurrent Q-networks for handling partial observability.

Step 3: Advanced Value-Based Methods Rainbow, C51-DQN, QR-DQN (2017) – Study ensemble methods and distributional RL.

IQN (2018) – Understand implicit quantile networks.

R2D2 (2019), NGU (2019) – Learn memory-efficient Q-learning.

Step 4: Policy Optimization A2C (2016), PPO (2016) – Study advantage actor-critic and trust region methods.

PPO-ICM, PPO-RND (2018) – Explore intrinsic motivation in RL.

SAC (2018) – Learn soft actor-critic for continuous action spaces.

Step 5: Modern Advances IMPALA (2021) – Understand scalable distributed RL.

Agent57 (2020) – Dive into the latest breakthroughs in exploration strategies.

Policy-based RL Algorithms

Directory	Reference Paper	Note
reinforce	Policy Gradient Methods for RL	*
reinforce_baseline	Policy Gradient Methods for RL	*
actor_critic	Actor-Critic Algorithms	*
a2c	Asynchronous Methods for Deep Reinforcement Learning | synchronous, deterministic variant of A3C	P
sac	Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning | Soft Actor-Critic for Discrete Action Settings	P *
ppo	Proximal Policy Optimization Algorithms	P
ppo_icm	Curiosity-driven Exploration by Self-supervised Prediction	P
ppo_rnd	Exploration by Random Network Distillation	P
impala	IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures	P

Value-based RL Algorithms

Directory	Reference Paper	Note
dqn	Human Level Control Through Deep Reinforcement Learning
double_dqn	Deep Reinforcement Learning with Double Q-learning
prioritized_dqn	Prioritized Experience Replay
drqn	Deep Recurrent Q-Learning for Partially Observable MDPs	*
r2d2	Recurrent Experience Replay in Distributed Reinforcement Learning	P
ngu	Never Give Up: Learning Directed Exploration Strategies	P *
agent57	Agent57: Outperforming the Atari Human Benchmark	P *

Distributional Q Learning Algorithms

Directory	Reference Paper	Note
c51_dqn	A Distributional Perspective on Reinforcement Learning
rainbow	Rainbow: Combining Improvements in Deep Reinforcement Learning
qr_dqn	Distributional Reinforcement Learning with Quantile Regression
iqn	Implicit Quantile Networks for Distributional Reinforcement Learning

Notes:

• P means support distributed training with multiple actors and a single learner running in parallel (only supports running on a single machine).
• * means only tested on Atari Pong or Breakout.

Code Structure

Author's Notes

• This project is for education and research purpose only. Where we focus on studying the individual algorithms rather than creating a standard library. If you're looking for a ready to use library for your productive application, this is probably the wrong place.
• Most agents only support episodic environment with discrete action space (except PPO which also supports continuous action space).
• Some code might not be optimal, especially the parts involving Python multiprocessing, as speed of code execution is not our main focus.
• Try our best to replicate the implementation for the original paper, but may change some hyper-parameters to support low budget setup. Also, the hyper-parameters and network architectures are not fine-tuned.
• For Atari games, we only use Pong or Breakout to test the agents, and we stop training once the agent have made some progress.
• We can't guarantee it's bug free. So bug report and pull request are welcome.

Quick Start

Please check the instructions in the QUICK_START.md file on how to setup the project.

Train Agents

Classic Control Tasks

• We maintain a list of environment names at gym_env.py module, by default it contains ['CartPole-v1', 'LunarLander-v2', 'MountainCar-v0', 'Acrobot-v1'].
• For some agents (like advanced DQN agents, most of the policy gradient agents except agents using curiosity-driven exploration), it's impossible to solve MountainCar due to the nature of the problem (sparse reward).

To run a agent on classic control problem, use the following command, replace the <agent_name> with the sub-directory name.

python3 -m deep_rl_box.<agent_name>.run_classic

# example of running DQN agents
python3 -m deep_rl_box.dqn.run_classic --environment_name=MountainCar-v0

python3 -m deep_rl_box.dqn.run_classic --environment_name=LunarLander-v2

Atari games

• By default, we uses gym NoFrameskip-v4 for Atari game, and we omit the need to include 'NoFrameskip' and version in the environment_name args, as it will be handled by create_atari_environment in the gym_env.py module.
• We don't scale the images before store into experience replay, as that will require 2-3x more RAM, we only scale them inside the model.forward() method.

To run a agent on Atari game, use the following command, replace the <agent_name> with the sub-directory name.

python3 -m deep_rl_box.<agent_name>.run_atari

# example of running DQN on Atari Pong and Breakout
python3 -m deep_rl_box.dqn.run_atari --environment_name=Pong

python3 -m deep_rl_box.dqn.run_atari --environment_name=Breakout

Distributed training with multiple actors and a single learner (on the same machine)

For agents that support distributed training, we can adjust the parameter num_actors to specify how many actors to run.

python3 -m deep_rl_box.ppo.run_classic --num_actors=8

The following is a high level overview of the distributed training architect. Where each actor has it's own copy of the neural network. And we use the multiprocessing.Queue to transfer the transitions between the actors and the leaner. We also use a shared dictionary to store the latest copy of the neural network's parameters, so the actors can get update it's local copy of the neural network later on.

# This will evenly distribute the actors on all GPUs
python3 -m deep_rl_box.ppo.run_atari --num_actors=16 --actors_on_gpu

# This will run all actors on CPU even if you have multiple GPUs
python3 -m deep_rl_box.ppo.run_atari --num_actors=16 --noactors_on_gpu

Evaluate Agents

Before you run the eval_agent module, make sure you have a valid checkpoint file for the specific agent and environment. By default, it will record a video of agent self-play at the recordings directory.

To run a agent on Atari game, use the following command, replace the <agent_name> with the sub-directory name.

python3 -m deep_rl_box.<agent_name>.eval_agent

# Example of load pre-trained PPO model on Breakout
python3 -m deep_rl_box.ppo.eval_agent --environment_name=Breakout --load_checkpoint_file=./checkpoints/PPO_Breakout_0.ckpt

Monitoring with Tensorboard

By default, both training, evaluation will log to Tensorboard at the runs directory. To disable this, use the option --nouse_tensorboard.

tensorboard --logdir=./runs

The classes for write logs to Tensorboard is implemented in trackers.py module.

• to improve performance, we only write logs at end of episode
• we separate training and evaluation logs
• if algorithm support parallel training, we separate actor, learner logs
• for agents that support parallel training, only log maximum of 8 actors, this is controlled by run_parallel_training_iterations in main_loop.py module

Measurements available on Tensorboard

performance(env_steps):

• the statistics are measured over env steps (or frames for Atari), if use frame_skip, it's counted after frame skip
• episode_return the non-discounted sum of raw rewards of current episode
• episode_steps the current episode length or steps
• num_episodes how many episodes have been conducted
• step_rate(second) step per seconds, per actors

agent_statistics(env_steps):

• the statistics are measured over env steps (or frames for Atari), if use frame_skip, it's counted after frame skip
• it'll log whatever is exposed in the agent's statistics property such as training loss, learning rate, discount, updates etc.
• for algorithm support distributed training (multiple actors), this is only the statistics for the actors.

learner_statistics(learner_steps):

• only available if the agent supports distributed training (multiple actors one learner)
• it'll log whatever is exposed in the learner's statistics property such as training loss, learning rate, discount, updates etc.
• to improve performance, it only logs every 100 learner steps

Add tags to Tensorboard

This could be handy if we want to compare different hyper parameter's performances or different runs with various seeds

python3 -m deep_rl_box.impala.run_classic --use_lstm --learning_rate=0.00045 --tag=LSTM-LR0.00045

Debug with environment screenshots

This could be handy if we want to see what's happening during the training, we can set the debug_screenshots_interval (measured over number of episode) to some value, and it'll add screenshots of the terminal state to Tensorboard.

# Example of creating terminal state screenshot every 100 episodes
python3 -m deep_rl_box.ppo_rnd.run_atari --environment_name=MontezumaRevenge --debug_screenshots_interval=100

Acknowledgments

This project is based on the work of DeepMind, specifically the following projects:

• DeepMind DQN Zoo
• DeepMind RLax

In addition, other reference projects from the community have been very helpful to us, including:

• Stable Baselines3
• OpenAI Spinning Up
• SEED RL
• TorchBeast

License

This project is licensed under the Apache License, Version 2.0 (the "License") see the LICENSE file for details

Citing our work

If you reference or use our project in your research, please cite:

@software{
  title = {{Deep RL Box}: A collections of Deep RL algorithms implemented with TensorFlow},
  author = {Sam Zamani},
  url = {https://github.com/samzmn/deep_rl_box},
  version = {0.0.1},
  year = {2025},
}