rlberry 0.0.2

An easy-to-use reinforcement learning library for research and education

A Reinforcement Learning Library for Research and Education

Try it on Google Colab!

---

Section	Description
Introduction	The philosophy of rlberry
Citation	How to cite this work
Getting started	A quick usage guide of rlberry
Installation	How to install rlberry
Documentation	A link to the documentation
Contributing	A guide for contributing

Introduction

The goal of rlberryto make Reinforcement Learning (RL) research and teaching easier. For that purpose, rlberry provides several interfaces that allow users to create their own experimental pipeline in a fast, clear and reproducible way. rlberry comprises the following major components and main features:

• An interface for agents, that

  • puts minimal constraints on the agent code (=> making it easy to include new algorithms and modify existing ones);

  • allows comparison between agents using a simple and unified evaluation interface (=> making it easy, for instance, to compare deep and "traditional" RL algorithms);

  • includes detailed documentation and comprehensible tutorial/examples (Jupyter Notebook) for each implemented agents (=> making it easy for education).

• An interface for environments, that

  • allows to create novel environments easily (=> making it possible to add your own environments);

  • totally adapts to OpenAI Gym (=> making it easy to use any existing environments from gym).

• An interface for rendering, that

  • provides simple and clear visualization of your experiments.

• Several important features include

  • a unified seeding mechanism: define only one global seed, from which all other seeds will inherit, enforcing independence of the random number generators;

  • an interface to Optuna that allows automatic hyperparameter optimization;

  • compatibility with Sacred that facilitates configuration, organization, logging and reproducing of computational experiments.

Citing rlberry

If you use rlberry in scientific publications, we would appreciate citations using the following Bibtex entry:

@misc{rlberry,
author = {Domingues, Omar Darwiche and ‪Flet-Berliac, Yannis and Leurent, Edouard and M{\'e}nard, Pierre and Shang, Xuedong and Valko, Michal},
title = {{rlberry - A Reinforcement Learning Library for Research and Education}},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/rlberry-py/rlberry}},
}

Getting started

We provide a handful of notebooks on Google colab as examples to show you how to use rlberry.

Content	Description	Link
Introduction to rlberry	How to create an agent, optimize its hyperparameters and compare to a baseline.
RL Experimental Pipeline	How to define a configuration, run experiments in parallel and save a config.json for reproducibility.

Compatibility with OpenAI Gym

If you want to use gym environments with rlberry, simply do the following:

from rlberry.envs import gym_make

# for example, let's take CartPole
env = gym_make('CartPole-v1')

This way, env behaves exactly the same as the gym environment, we simply replace the seeding function by env.reseed(), which ensures unified seeding and reproducibility when using rlberry.

Seeding

In rlberry, only one global seed is defined, and all the random number generators used by the agents and environments inherit from this seed, ensuring reproducibility and independence between the generators (see NumPy SeedSequence).

Example:

import rlberry.seeding as seeding

seeding.set_global_seed(seed=123)

# From now on, no more seeds are defined by the user, and all the results are reproducible.
...

# If you need a random number generator (rng), call:
rng = seeding.get_rng()   

# which gives a numpy Generator (https://numpy.org/doc/stable/reference/random/generator.html) 
# that is independent of all the previous generators created by seeding.get_rng()
rng.integers(5)
rng.normal()
# etc

Installation

Cloning & creating virtual environment

It is suggested to create a virtual environment using Anaconda or Miniconda:

git clone https://github.com/rlberry-py/rlberry.git
conda create -n rlberry python=3.7

Basic installation

Install without heavy libraries (e.g. pytorch):

conda activate rlberry
pip install -e .

Full installation

Install with all features:

conda activate rlberry
pip install -e .[full]

which includes:

• Numba for just-in-time compilation of algorithms based on dynamic programming
• PyTorch for Deep RL agents
• Optuna for hyperparameter optimization
• Sacred for handling experiment configurations
• ffmpeg-python for saving videos
• PyOpenGL for more rendering options

Tests

To run tests, install test dependencies with pip install -e .[test] and run pytest. To run tests with coverage, install test dependencies and run bash run_testscov.sh. See coverage report in cov_html/index.html.

Documentation

The documentation is under construction and will be available here.

Contributing

Want to contribute to rlberry? Please check our contribution guidelines. A list of interesting TODO's will be available soon. If you want to add any new agents or environments, do not hesitate to open an issue!

Implementation notes

• When inheriting from the Agent class, make sure to call Agent.__init__(self, env, **kwargs) using **kwargs in case new features are added to the base class, and to make sure that copy_env and reseed_env are always an option to any agent.

Infos, errors and warnings are printed using the logging library.

• From gym to rlberry:
  • reseed (rlberry) should be called instead of seed (gym). seed keeps compatilibity with gym, whereas reseed uses the unified seeding mechanism of rlberry.