reinforced-lib 0.2.0

Reinforcement learning library

Reinforced-lib: Reinforcement learning library

Overview

Reinforced-lib is a Python library designed to support research and prototyping using reinforcement learning (RL) algorithms. The library can serve as a simple solution with ready to use RL workflows as well as an expandable framework with programmable behaviour. Thanks to the functional implementation of the library's core, we can provide full access to the JAX's jit functionality, which boosts the agent's performance significantly.

Installation

You can install the latest released version of Reinforced-lib from PyPI via:

pip install reinforced-lib

You can also download source code and install the development dependencies if you want to build the documentation locally:

git clone [e-mail]:m-wojnar/reinforced-lib.git
cd reinforced-lib
pip3 install ".[dev]"

Example code

from reinforced_lib import RLib
from reinforced_lib.agents import ThompsonSampling
from reinforced_lib.exts import IEEE_802_11_ax

import gym


rlib = RLib(
  agent_type=ThompsonSampling,
  ext_type=IEEE_802_11_ax
)

env = gym.make('WifiSimulator-v1')
env_state, _ = env.reset()

terminated = False
while not terminated:
  action = rlib.sample(**env_state)
  env_state, reward, terminated, *_ = env.step(action)

Integrated IEEE 802.11ax support

Library design is distinctly influenced by the desire to support research in Wi-Fi. It can be a tool for researchers to optimize the Wi-Fi protocols with built-in RL algorithms and provided IEEE 802.11ax environment extension.

We also provide simple ns-3 simulation and RL-based rate adaptation manager for the IEEE 802.11ax standard in examples.

Modular architecture

Reinforced-lib can be well characterized by it's modular architecture which makes the library flexible, universal, and easy-to-use. Its key parts are placed in separate modules and connected in a standardized way to provide versatility and the possibility to extend individual modules in the future. Nevertheless, Reinforced-lib is a single piece of software that can be easily used, thanks to the topmost module, which ensures a simple and common interface for all agents.

The API module

The API module is the top layer of the library; it exposes a simple and intuitive interface that makes the library easy to use. There are several important methods, one of them is responsible for creating a new agent. Another takes the observations from the environment as input, updates the state of the agent, and returns the next action proposed by the agent. The last two methods are used to persist the state of the agent by storing it in memory.

The extensions module

The Extensions module consists of containers with domain-specific knowledge and ensures the proper use of universal agents implemented in Reinforced-lib. If a specific problem is implemented in the form of an extension, the module infers and provides the appropriate data to the agent, and at the same time requires adequate, corresponding values from the user.

The agents module

The Agents module is a collection of universal algorithms, which are called "agents" in RL community. Each agent has a similar API to communicate with the Extensions module, which ensures its versatility and expandability. In this release of Reinforced-lib we focused on the multi-armed bandit problem, hence the imlemented agents are related to this task.

The logging module

The Logging module is responsible for collecting data from other modules and observing their state in real time. It also has great potential in using the library to create new RL agents - it can be used to develop, evaluate, and debug new agents by observing decisions they make; record and visualize how environment state changes in time; or provide a simple way to obtain a training summary, metrics, and logs.

Citing Reinforced-lib

To cite this repository:

@software{reinforcedlib2022,
  author = {Maksymilian Wojnar and Wojciech Ciężobka},
  title = {{R}einforced-lib: {R}einforcement learning library},
  url = {http://github.com/m-wojnar/reinforced-lib},
  year = {2022},
}