Skip to main content

An offline deep reinforcement learning library

Project description

d3rlpy: An offline deep reinforcement learning library

test build Documentation Status codecov Maintainability Gitter MIT

d3rlpy is an offline deep reinforcement learning library for practitioners and researchers.

import d3rlpy

dataset, env = d3rlpy.datasets.get_dataset("hopper-medium-v0")

# prepare algorithm
sac = d3rlpy.algos.SAC()

# train offline
sac.fit(dataset, n_steps=1000000)

# train online
sac.fit_online(env, n_steps=1000000)

# ready to control
actions = sac.predict(x)

Documentation: https://d3rlpy.readthedocs.io

key features

:zap: Most Practical RL Library Ever

  • offline RL: d3rlpy supports state-of-the-art offline RL algorithms. Offline RL is extremely powerful when the online interaction is not feasible during training (e.g. robotics, medical).
  • online RL: d3rlpy also supports conventional state-of-the-art online training algorithms without any compromising, which means that you can solve any kinds of RL problems only with d3rlpy.
  • advanced engineering: d3rlpy is designed to implement the faster and efficient training algorithms. For example, you can train Atari environments with x4 less memory space and as fast as the fastest RL library.

:beginner: Easy-To-Use API

  • zero-knowledge of DL library: d3rlpy provides many state-of-the-art algorithms through intuitive APIs. You can become a RL engineer even without knowing how to use deep learning libraries.
  • scikit-learn compatibility: d3rlpy is not only easy, but also completely compatible with scikit-learn API, which means that you can maximize your productivity with the useful scikit-learn's utilities.

:rocket: Beyond State-Of-The-Art

  • distributional Q function: d3rlpy is the first library that supports distributional Q functions in the all algorithms. The distributional Q function is known as the very powerful method to achieve the state-of-the-performance.
  • many tweek options: d3rlpy is also the first to support N-step TD backup and ensemble value functions in the all algorithms, which lead you to the place no one ever reached yet.

installation

d3rlpy supports Linux, macOS and Windows.

PyPI (recommended)

PyPI version PyPI - Downloads

$ pip install d3rlpy

Anaconda

Anaconda-Server Badge Anaconda-Server Badge Anaconda-Server Badge

$ conda install -c conda-forge d3rlpy

Docker

Docker Pulls

$ docker run -it --gpus all --name d3rlpy takuseno/d3rlpy:latest bash

supported algorithms

algorithm discrete control continuous control offline RL?
Behavior Cloning (supervised learning) :white_check_mark: :white_check_mark:
Deep Q-Network (DQN) :white_check_mark: :no_entry:
Double DQN :white_check_mark: :no_entry:
Deep Deterministic Policy Gradients (DDPG) :no_entry: :white_check_mark:
Twin Delayed Deep Deterministic Policy Gradients (TD3) :no_entry: :white_check_mark:
Soft Actor-Critic (SAC) :white_check_mark: :white_check_mark:
Batch Constrained Q-learning (BCQ) :white_check_mark: :white_check_mark: :white_check_mark:
Bootstrapping Error Accumulation Reduction (BEAR) :no_entry: :white_check_mark: :white_check_mark:
Advantage-Weighted Regression (AWR) :white_check_mark: :white_check_mark: :white_check_mark:
Conservative Q-Learning (CQL) (recommended) :white_check_mark: :white_check_mark: :white_check_mark:
Advantage Weighted Actor-Critic (AWAC) :no_entry: :white_check_mark: :white_check_mark:
Critic Reguralized Regression (CRR) :no_entry: :white_check_mark: :white_check_mark:
Policy in Latent Action Space (PLAS) :no_entry: :white_check_mark: :white_check_mark:
TD3+BC :no_entry: :white_check_mark: :white_check_mark:

supported Q functions

other features

Basically, all features are available with every algorithm.

  • <input type="checkbox" checked="" disabled="" /> evaluation metrics in a scikit-learn scorer function style
  • <input type="checkbox" checked="" disabled="" /> export greedy-policy as TorchScript or ONNX
  • <input type="checkbox" checked="" disabled="" /> parallel cross validation with multiple GPU

experimental features

examples

MuJoCo

import d3rlpy

# prepare dataset
dataset, env = d3rlpy.datasets.get_d4rl('hopper-medium-v0')

# prepare algorithm
cql = d3rlpy.algos.CQL(use_gpu=True)

# train
cql.fit(dataset,
        eval_episodes=dataset,
        n_epochs=100,
        scorers={
            'environment': d3rlpy.metrics.evaluate_on_environment(env),
            'td_error': d3rlpy.metrics.td_error_scorer
        })

See more datasets at d4rl.

Atari 2600

import d3rlpy
from sklearn.model_selection import train_test_split

# prepare dataset
dataset, env = d3rlpy.datasets.get_atari('breakout-expert-v0')

# split dataset
train_episodes, test_episodes = train_test_split(dataset, test_size=0.1)

# prepare algorithm
cql = d3rlpy.algos.DiscreteCQL(n_frames=4, q_func_factory='qr', scaler='pixel', use_gpu=True)

# start training
cql.fit(train_episodes,
        eval_episodes=test_episodes,
        n_epochs=100,
        scorers={
            'environment': d3rlpy.metrics.evaluate_on_environment(env),
            'td_error': d3rlpy.metrics.td_error_scorer
        })

See more Atari datasets at d4rl-atari.

PyBullet

import d3rlpy

# prepare dataset
dataset, env = d3rlpy.datasets.get_pybullet('hopper-bullet-mixed-v0')

# prepare algorithm
cql = d3rlpy.algos.CQL(use_gpu=True)

# start training
cql.fit(dataset,
        eval_episodes=dataset,
        n_epochs=100,
        scorers={
            'environment': d3rlpy.metrics.evaluate_on_environment(env),
            'td_error': d3rlpy.metrics.td_error_scorer
        })

See more PyBullet datasets at d4rl-pybullet.

Online Training

import d3rlpy
import gym

# prepare environment
env = gym.make('HopperBulletEnv-v0')
eval_env = gym.make('HopperBulletEnv-v0')

# prepare algorithm
sac = d3rlpy.algos.SAC(use_gpu=True)

# prepare replay buffer
buffer = d3rlpy.online.buffers.ReplayBuffer(maxlen=1000000, env=env)

# start training
sac.fit_online(env, buffer, n_steps=1000000, eval_env=eval_env)

tutorials

Try a cartpole example on Google Colaboratory!

  • offline RL tutorial: Open In Colab
  • online RL tutorial: Open In Colab

contributions

Any kind of contribution to d3rlpy would be highly appreciated! Please check the contribution guide.

The release planning can be checked at milestones.

community

Channel Link
Chat Gitter
Issues GitHub Issues
Discussion GitHub Discussions

family projects

Project Description
d4rl-pybullet An offline RL datasets of PyBullet tasks
d4rl-atari A d4rl-style library of Google's Atari 2600 datasets
MINERVA An out-of-the-box GUI tool for offline RL

citation

@misc{seno2020d3rlpy,
  author = {Takuma Seno},
  title = {d3rlpy: An offline deep reinforcement library},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/takuseno/d3rlpy}}
}

acknowledgement

This work is supported by Information-technology Promotion Agency, Japan (IPA), Exploratory IT Human Resources Project (MITOU Program) in the fiscal year 2020.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for d3rlpy, version 0.91
Filename, size File type Python version Upload date Hashes
Filename, size d3rlpy-0.91-cp36-cp36m-macosx_10_14_x86_64.whl (404.3 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size d3rlpy-0.91-cp36-cp36m-manylinux1_x86_64.whl (1.0 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size d3rlpy-0.91-cp36-cp36m-win_amd64.whl (355.6 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size d3rlpy-0.91-cp37-cp37m-macosx_10_14_x86_64.whl (404.4 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size d3rlpy-0.91-cp37-cp37m-manylinux1_x86_64.whl (1.0 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size d3rlpy-0.91-cp37-cp37m-win_amd64.whl (355.6 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size d3rlpy-0.91-cp38-cp38-macosx_10_14_x86_64.whl (406.0 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size d3rlpy-0.91-cp38-cp38-manylinux1_x86_64.whl (1.1 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size d3rlpy-0.91-cp38-cp38-win_amd64.whl (362.3 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size d3rlpy-0.91.tar.gz (337.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page