flashrl 0.0.4

Fast reinforcement learning 💨

flashrl

RL in seconds 💨 with ~200 lines of code (+ ~150 per env) 🤓

🛠️ pip install flashrl, or if you want to modify envs, clone the repo and pip install -r requirements.txt

Quick Start 🚀

0. If installed via clone, compile envs: python setup.py build_ext --inplace
1. Train: python train.py
2. See the magic unfold in the terminal 🪄

Usage 💡

flashrl will always be short: Read the code (+paste into ChatGPT) to understand it!

Here's a minimal example to get you going:

flashrl uses a Learner that holds an env and a model (default: LSTMPolicy)

import flashrl as frl

learn = frl.Learner(env=frl.envs.Pong(n_agents=2**14))
curves = learn.fit(40, steps=16, pbar_desc='done')
frl.print_ascii_curve(curves['loss'], label='loss')
frl.render_ascii(learn, fps=10)
learn.env.close()

.fit triggers RL with

• 40 iterations...
• ...16 steps per iteration...
• ...in Pong holding 2**14=16384 agents

resulting in training with (40 * 16 * 16384=)~10 million steps!

Click here, to read a tiny doc 📑

.fit takes the arguments

• iters: Number of iterations
• steps: Number of steps in rollout
• pbar_desc: Progress bar description (default: 'reward')
• log: If True, tensorboard logging is enabled
  • run tensorboard --logdir=runsand visit http://localhost:6006 in the browser!
• lr, anneal_lr, target_fl + all args of ppo: Hyperparameters

Take a look at train.py to see how to use the utils-functions

• print_ascii_curve: Visualizes the loss across the iters
• render_ascii: Shows data of the last rollout in the terminal
• render_gif: Shows the same, saved as a GIF
• print_table: Shows a table of values, acts, logprobs, reward and dones of the last rollout

Environments 🕹️

Each env is one Cython(=.pyx) file in flashrl/envs. That's it!

To add custom envs, use grid.pyx, pong.pyx or multigrid.pyx as a template:

• grid.pyx for single-agent envs
• pong.pyx for 1 vs. 1 agent envs (AlphaZero-style)
• multigrid.pyx for multi-agent envs

Grid	Pong	MultiGrid
Agent must reach goal	Good old pong (1 vs. 1)	Agent must reach goal first

Acknowledgements 🙌

I want to thank

• Joseph Suarez for open sourcing RL envs in C(ython)! Star PufferLib ⭐
• Costa Huang for open sourcing high-quality single-file RL code! Star cleanrl ⭐

and last but not least...