easyagents 1.2.2

reinforcement learning for practitioners.

Reinforcement Learning for Practitioners (v1.2, 19Q4)

Status: under active development, breaking changes may occur. Release notes.

EasyAgents is a high level reinforcement learning api focusing on ease of use and simplicity. Written in Python and running on top of established reinforcement learning libraries like tf-Agents, tensorforce or keras-rl. Environments are implemented in OpenAI gym.

In collaboration with Oliver Zeigermann.

Features

• provides the same, simple api across all libraries. Thus you can easily switch between different implementations and you don't have to learn for each of them a new api.
• to create and run any algorithm you only need 2 lines of code, all the parameters are named consistently across all algorithms.
• supports a broad set of different algorithms
• runs inside jupyter notebooks as well as stand-alone, easy to install requiring only a single 'pip install'.
• easy to understand, ready-made plots and logs to investigate the algorithms and environments behaviour

Try it on colab

• 1. Introduction (CartPole on colab): training, plotting, switching algorithms & backends. based on the classic reinforcement learning example balancing a stick on a cart.
• 2. Next steps & backend switching (Orso on colab): custom training, creating a movie & switching backends. gym environment based on a routing problem.
• 3. Creating your own environment (LineWorld on colab): implement a gym environment from scratch, workshop example.
• 4. Logging, seeding & plot clearing (on colab): Investigate how an agents api and how it interacts with the gym environment; how to set seeds; controlling jupyter output cell clearing

Scenario: simple

from easyagents.agents import PpoAgent
from easyagents.callbacks import plot

ppoAgent = PpoAgent('CartPole-v0')
ppoAgent.train([plot.State(), plot.Loss(), plot.Rewards()])

Scenario: more detailed

from easyagents.agents import PpoAgent
from easyagents.callbacks import plot

ppoAgent = PpoAgent( 'Orso-v1',fc_layers=(500,500,500))
ppoAgent.train([plot.State(), plot.Loss(), plot.Rewards(), plot.Actions(), 
                plot.StepRewards(), plot.Steps(), plot.ToMovie()], 
                learning_rate = 0.0001, num_iterations = 500, max_steps_per_episode=50 )

Available Algorithms and Backends

algorithm	tf-Agents	tensorforce	keras-rl	easyagents class name
CEM	not available	not available	yes	CemAgent
Dqn	yes	yes	yes	DqnAgent
Double Dqn	open	not available	yes	DoubleDqnAgent
Dueling Dqn	not available	not available	yes	DuelingDqnAgent
Ppo	yes	yes	not available	PpoAgent
Random	yes	yes	not available	RandomAgent
REINFORCE	yes	yes	not available	ReinforceAgent
SAC	preview	not available	not available	SacAgent

[191001]

• if you are interested in other algorithms, backends or hyperparameters let us know by creating an issue. We'll try our best to support you.
• keras-rl is not compatible with tensorflow eager execution mode. Thus keras-rl based agents should run in a different python / jupyter notebook instance than tf-agents or tensorforce based agents.

Guiding Principles

• easily train, evaluate & debug policies for (you own) gym environment over "designing new algorithms"
• simple & consistent over "flexible & powerful"
• inspired by keras:
  • same api across all algorithms
  • support different implementations of the same algorithm
  • extensible (pluggable backends, plots & training schemes)

Installation

Install from pypi using pip:

pip install easyagents

Documentation

release notes, class diagram

EasyAgents may not be ideal if

• you would like to leverage implementation specific advantages of an algorithm
• you want to do distributed or in parallel reinforcement learning

Note

• This repository is under active development and in an early stage. Thus any- and everything may (and probably should) change.
• If you have any difficulties in installing or using easyagents please let us know by creating an issue. We'll try our best to help you.
• Any ideas, help, suggestions, comments etc in python / open source development / reinforcement learning / whatever are more than welcome. Thanks a lot in advance.