Skip to main content

ReplayBuffer for Reinforcement Learning written by C++ and Cython

Project description

img img img img img

img

Overview

cpprb is a python (CPython) module providing replay buffer classes for reinforcement learning.

Major target users are researchers and library developers.

You can build your own reinforcement learning algorithms together with your favorite deep learning library (e.g. TensorFlow, PyTorch).

cpprb forcuses speed, flexibility, and memory efficiency.

By utilizing Cython, complicated calculations (e.g. segment tree for prioritized experience replay) are offloaded onto C++. (The name cpprb comes from "C++ Replay Buffer".)

In terms of API, initially cpprb referred to OpenAI Baselines' implementation. In the current version, cpprb has much more flexibility. Any NumPy compatible types of any numbers of values can be stored (as long as memory capacity is sufficient). For example, you can store the next action and the next next observation, too.

Installation

cpprb requires following softwares before installation.

  • C++17 compiler (for installation from source)
  • Python 3
  • pip

Cuurently, clang, which is a default Xcode C/C++ compiler at Apple macOS, cannot compile cpprb.

If you are macOS user, you need to install GCC and set environment values of CC and CXX to g++, or just use virtual environment (e.g. Docker).

Step by step installation is described here.

Additionally, here are user's good feedbacks for installation at macOS and Ubuntu. (Thanks!)

Install from PyPI (Recommended)

The following command installs cpprb together with other dependancies.

pip install cpprb

Depending on your environment, you might need sudo or --user flag for installation.

On supported platflorms (Linux x86-64 and Windows amd64), binary packages are hosted on PyPI can be used, so that you don't need C++ compiler.

If you have trouble to install from binary, you can fall back to source installation to passk --no-binary option to the above pip command.

Currently, no other platforms, such as macOS, and 32bit or arm-architectured Linux and Windows, cannot install from binary, and need to compile by yourself. Please be patient, we will plan to support wider platforms in future.

Install from source code

First, download source code manually or clone the repository;

git clone https://gitlab.com/ymd_h/cpprb.git

Then you can install same way;

cd cpprb
pip install .

For this installation, you need to convert extended Python (.pyx) to C++ (.cpp) during installation, it takes longer time than installation from PyPI.

Usage

Here is a simple example for storing standard environment (aka. "obs", "act", "rew", "nextobs", and "done").

from cpprb import ReplayBuffer

buffer_size = 256
obs_shape = 3
act_dim = 1
rb = ReplayBuffer(buffer_size,
		  env_dict ={"obs": {"shape": obs_shape},
			     "act": {"shape": act_dim},
			     "rew": {},
			     "next_obs": {"shape": obs_shape},
			     "done": {}})

obs = np.ones(shape=(obs_shape))
act = np.ones(shape=(act_dim))
rew = 0
next_obs = np.ones(shape=(obs_shape))
done = 0

for i in range(500):
    rb.add(obs=obs,act=act,rew=rew,next_obs=next_obs,done=done)


batch_size = 32
sample = rb.sample(batch_size)
# sample is a dictionary whose keys are 'obs', 'act', 'rew', 'next_obs', and 'done'

Flexible environment values are defined by env_dict when buffer creation.

Since stored values have flexible name, you have to pass to ReplayBuffer.add member by keyword.

Features

cpprb provides buffer classes for building following algorithms.

Algorithms cpprb class Paper
Experience Replay `ReplayBuffer` [L. J. Lin](https://link.springer.com/article/10.1007/BF00992699)
Prioritized Experience Replay `PrioritizedReplayBuffer` [T. Schaul et. al.](https://arxiv.org/abs/1511.05952)
Multi-step Learning `ReplayBuffer`, `PrioritizedReplayBuffer`  

cpprb features and its usage are described at following pages:

Contributing to cpprb

Any contribution are very welcome!

Making Community Larger

Bigger commumity makes development more active and improve cpprb.

  • Star this repository (and/or GitHub Mirror)
  • Publish your code using cpprb
  • Share this repository to your friend and/or followers.

Report Issue

When you have any problems or requests, you can check issues on GitLab.com. If you still cannot find any information, you can open your own issue.

Merge Request (Pull Request)

cpprb follows local rules:

  • Branch Name
    • "HotFix***" for bug fix
    • "Feature***" for new feature implementation
  • docstring
  • Unit Test
    • Put test code under "test/" directory
    • Can test by python -m unittest <Your Test Code> command
    • Continuous Integration on GitLab CI configured by .gitlab-ci.yaml
  • Open an issue and associate it to Merge Request

Step by step instruction for beginners is described at here.

Links

cpprb sites

cpprb users' repositories

Lisence

cpprb is available under MIT lisence.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cpprb-8.4.7.tar.gz (309.7 kB view details)

Uploaded Source

Built Distributions

cpprb-8.4.7-cp38-cp38-win_amd64.whl (245.0 kB view details)

Uploaded CPython 3.8Windows x86-64

cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.12+ x86-64

cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.8

cpprb-8.4.7-cp37-cp37m-win_amd64.whl (237.4 kB view details)

Uploaded CPython 3.7mWindows x86-64

cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.12+ x86-64

cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.7m

cpprb-8.4.7-cp36-cp36m-win_amd64.whl (237.4 kB view details)

Uploaded CPython 3.6mWindows x86-64

cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.12+ x86-64

cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.6m

cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl (1.4 MB view details)

Uploaded CPython 3.5mmanylinux: glibc 2.12+ x86-64

cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl (1.4 MB view details)

Uploaded CPython 3.5m

File details

Details for the file cpprb-8.4.7.tar.gz.

File metadata

  • Download URL: cpprb-8.4.7.tar.gz
  • Upload date:
  • Size: 309.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7.tar.gz
Algorithm Hash digest
SHA256 45b4798b974113983328e60d3187a3b80b66520b435f67cfb048ea04ec447b85
MD5 4ec48eef0f33b6405b7797358e446d1a
BLAKE2b-256 60d2d6b362596cbe6409cc73795536ac399d53d4d40652dacc5f8fcea60e0c83

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 245.0 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.2

File hashes

Hashes for cpprb-8.4.7-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1f604ed50c78ed994787172f823d723ade6c1b7d56bc4a2df3f896327352a5fd
MD5 22ef2a5abc50ebc85e693396b193880c
BLAKE2b-256 65ab20de796fa73df045339c8353d63423361eaf6b3e7229b86d0820807989bd

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e83cdd4c2294f7729f505aa027146f3306f8cc9c256fb8bc0401a5486c94ff89
MD5 29aba3b6795007b0d84abd66de58d526
BLAKE2b-256 34f1e68803543da2143abcd5acae09c407ea1c414193cd90e839747080af5aa1

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 76a60dfe202f4fadd7d835f59bfd7ea9b3c75c0b758235af1fe69312592de88a
MD5 0f87ea1e9ccfbd8669d785b5b381a7ce
BLAKE2b-256 7000242b02f6729f12e376673c738481dfb0d2eb4a070e62c974afaae2f5ec3b

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 237.4 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 98847ae4651db04bcfffd4a1b216e35c9a9975befe86d47c5cde810e054adb93
MD5 0fc161597439a477785cd6ab6ebd8a14
BLAKE2b-256 a3a4d36e027590809291a17d4690a0d733a3fdfa0978c0fe832ab5cca2e96d2b

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 e5010ad443d102286fa3454effa3c55106e7d1722e1848efcb1a1e3b6199b258
MD5 8027066b821f0f6e549a57d1887a7ba1
BLAKE2b-256 0f4581dc7644443debf2317e941492614a2aaef4741bd82d05a67f4d63f20e98

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3b7628431b6b9f03d33d8b1851fc0099dcbfb3789952a99d971731b3b71100df
MD5 44136a339d7fcd75624070d145ac0463
BLAKE2b-256 89adba8485368e248bca4e618a6acd76a4759c1beeff7bfa666201832cdc6f70

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 237.4 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.8

File hashes

Hashes for cpprb-8.4.7-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 5818d5674f030812c8925449974931f6a715ebdf90be44f377ca23f911354192
MD5 d33284037d4a0dbf98caed76c628c5f5
BLAKE2b-256 5291ff6288d5f1e022b25e5283a5149ccba62493848ac04e3e726f3a42e5c8ca

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 6d093fc1edd02c165f5aee7ab2c66dec9887a04b10b664ebcbe429df05615c50
MD5 8f12ea1982c7f4d4b05ff2da9c6704e4
BLAKE2b-256 7cf7501de4d7eeb93eb405847506d61e0c6c0f0f865fa97257b3f2a315508cff

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 524ebe6dd60b52ef37044c3b5883dd545bab50ed5c50cc483e0bfc9ea80278b2
MD5 44114c066d2ab40e2bf88ace8a6ea2a0
BLAKE2b-256 a37d2759aa9283f8fa7c43b67c20d10177d742fd2e76701efa89e2aa94707834

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.5m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 f779e985400b4fc48e6f26df6057c5f96fc54de3b5f587d2ef430d228a3c4193
MD5 8ffcd7a2e6a08de9f61c870d969d32ad
BLAKE2b-256 db919e7766b6717bd829fc4ddca71f2291bdd6680d6408841e323052fbc11a62

See more details on using hashes here.

File details

Details for the file cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d77c693a24e8d0af1e1e480e69447ad333e32a3481b72494da99628d086245c9
MD5 48a43dbb6c2b29991fcae0f8c4167752
BLAKE2b-256 8e9707e7a01b44dde555f236c93c05756d9b36120404cacceab9397ec1db5827

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page