Skip to main content

ReplayBuffer for Reinforcement Learning written by C++ and Cython

Project description

img img img img img

img

Overview

cpprb is a python (CPython) module providing replay buffer classes for reinforcement learning.

Major target users are researchers and library developers.

You can build your own reinforcement learning algorithms together with your favorite deep learning library (e.g. TensorFlow, PyTorch).

cpprb forcuses speed, flexibility, and memory efficiency.

By utilizing Cython, complicated calculations (e.g. segment tree for prioritized experience replay) are offloaded onto C++. (The name cpprb comes from "C++ Replay Buffer".)

In terms of API, initially cpprb referred to OpenAI Baselines' implementation. In the current version, cpprb has much more flexibility. Any NumPy compatible types of any numbers of values can be stored (as long as memory capacity is sufficient). For example, you can store the next action and the next next observation, too.

Installation

cpprb requires following softwares before installation.

  • C++17 compiler (for installation from source)
  • Python 3
  • pip

Cuurently, clang, which is a default Xcode C/C++ compiler at Apple macOS, cannot compile cpprb.

If you are macOS user, you need to install GCC and set environment values of CC and CXX to g++, or just use virtual environment (e.g. Docker).

Step by step installation is described here.

Additionally, here are user's good feedbacks for installation at macOS and Ubuntu. (Thanks!)

Install from PyPI (Recommended)

The following command installs cpprb together with other dependancies.

pip install cpprb

Depending on your environment, you might need sudo or --user flag for installation.

On supported platflorms (Linux x86-64 and Windows amd64), binary packages are hosted on PyPI can be used, so that you don't need C++ compiler.

If you have trouble to install from binary, you can fall back to source installation to passk --no-binary option to the above pip command.

Currently, no other platforms, such as macOS, and 32bit or arm-architectured Linux and Windows, cannot install from binary, and need to compile by yourself. Please be patient, we will plan to support wider platforms in future.

Install from source code

First, download source code manually or clone the repository;

git clone https://gitlab.com/ymd_h/cpprb.git

Then you can install same way;

cd cpprb
pip install .

For this installation, you need to convert extended Python (.pyx) to C++ (.cpp) during installation, it takes longer time than installation from PyPI.

Usage

Basic Usage

Basic usage is following step;

  1. Create replay buffer (ReplayBuffer.__init__)
  2. Add transitions (ReplayBuffer.add)
    1. Reset at episode end (ReplayBuffer.on_episode_end)
  3. Sample transitions (ReplayBuffer.sample)

Example Code

Here is a simple example for storing standard environment (aka. "obs", "act", "rew", "next_obs", and "done").

from cpprb import ReplayBuffer

buffer_size = 256
obs_shape = 3
act_dim = 1
rb = ReplayBuffer(buffer_size,
		  env_dict ={"obs": {"shape": obs_shape},
			     "act": {"shape": act_dim},
			     "rew": {},
			     "next_obs": {"shape": obs_shape},
			     "done": {}})

obs = np.ones(shape=(obs_shape))
act = np.ones(shape=(act_dim))
rew = 0
next_obs = np.ones(shape=(obs_shape))
done = 0

for i in range(500):
    rb.add(obs=obs,act=act,rew=rew,next_obs=next_obs,done=done)

    if done:
	# Together with resetting environment, call ReplayBuffer.on_episode_end()
	rb.on_episode_end()

batch_size = 32
sample = rb.sample(batch_size)
# sample is a dictionary whose keys are 'obs', 'act', 'rew', 'next_obs', and 'done'

Construction Parameters

(See also API reference)

Name Type Optional Discription
`size` `int` No Buffer size
`env_dict` `dict` Yes (but unusable) Environment definition (See [here](https://ymd_h.gitlab.io/cpprb/features/flexible_environment/))
`next_of` `str` or array-like of `str` Yes Memory compression (See [here](https://ymd_h.gitlab.io/cpprb/features/memory_compression/))
`stack_compress` `str` or array-like of `str` Yes Memory compression (See [here](https://ymd_h.gitlab.io/cpprb/features/memory_compression/))
`default_dtype` `numpy.dtype` Yes Fall back data type
`Nstep` `dict` Yes Nstep configuration (See [here](https://ymd_h.gitlab.io/cpprb/features/nstep/))
`mmap_prefix` `str` Yes mmap file prefix (See [here](https://ymd_h.gitlab.io/cpprb/features/mmap/))

Notes

Flexible environment values are defined by env_dict when buffer creation. The detail is described at document.

Since stored values have flexible name, you have to pass to ReplayBuffer.add member by keyword.

Features

cpprb provides buffer classes for building following algorithms.

Algorithms cpprb class Paper
Experience Replay `ReplayBuffer` [L. J. Lin](https://link.springer.com/article/10.1007/BF00992699)
[Prioritized Experience Replay](https://ymd_h.gitlab.io/cpprb/features/per/) `PrioritizedReplayBuffer` [T. Schaul et. al.](https://arxiv.org/abs/1511.05952)
[Multi-step (Nstep) Learning](https://ymd_h.gitlab.io/cpprb/features/nstep/) `ReplayBuffer`, `PrioritizedReplayBuffer`  

cpprb features and its usage are described at following pages:

Contributing to cpprb

Any contribution are very welcome!

Making Community Larger

Bigger commumity makes development more active and improve cpprb.

  • Star this repository (and/or GitHub Mirror)
  • Publish your code using cpprb
  • Share this repository to your friend and/or followers.

Report Issue

When you have any problems or requests, you can check issues on GitLab.com. If you still cannot find any information, you can open your own issue.

Merge Request (Pull Request)

cpprb follows local rules:

  • Branch Name
    • "HotFix***" for bug fix
    • "Feature***" for new feature implementation
  • docstring
  • Unit Test
    • Put test code under "test/" directory
    • Can test by python -m unittest <Your Test Code> command
    • Continuous Integration on GitLab CI configured by .gitlab-ci.yaml
  • Open an issue and associate it to Merge Request

Step by step instruction for beginners is described at here.

Links

cpprb sites

cpprb users' repositories

Lisence

cpprb is available under MIT lisence.

MIT License

Copyright (c) 2020 Yamada Hiroyuki

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Project details


Release history Release notifications | RSS feed

This version

9.2.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cpprb-9.2.1.tar.gz (329.9 kB view details)

Uploaded Source

Built Distributions

cpprb-9.2.1-cp38-cp38-win_amd64.whl (244.7 kB view details)

Uploaded CPython 3.8Windows x86-64

cpprb-9.2.1-cp38-cp38-manylinux2010_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.12+ x86-64

cpprb-9.2.1-cp38-cp38-manylinux1_x86_64.whl (1.8 MB view details)

Uploaded CPython 3.8

cpprb-9.2.1-cp37-cp37m-win_amd64.whl (236.9 kB view details)

Uploaded CPython 3.7mWindows x86-64

cpprb-9.2.1-cp37-cp37m-manylinux2010_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.12+ x86-64

cpprb-9.2.1-cp37-cp37m-manylinux1_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.7m

cpprb-9.2.1-cp36-cp36m-win_amd64.whl (236.8 kB view details)

Uploaded CPython 3.6mWindows x86-64

cpprb-9.2.1-cp36-cp36m-manylinux2010_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.12+ x86-64

cpprb-9.2.1-cp36-cp36m-manylinux1_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.6m

cpprb-9.2.1-cp35-cp35m-manylinux2010_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.5mmanylinux: glibc 2.12+ x86-64

cpprb-9.2.1-cp35-cp35m-manylinux1_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.5m

File details

Details for the file cpprb-9.2.1.tar.gz.

File metadata

  • Download URL: cpprb-9.2.1.tar.gz
  • Upload date:
  • Size: 329.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1.tar.gz
Algorithm Hash digest
SHA256 2e50dde0a591c7c0bc1895355096bd80ac360920c60a36d2887cae57bb3c9abd
MD5 35f9fa8301cbce08b3d650b2f7f6f535
BLAKE2b-256 e94b0b7a868cab38f9ddba22f6b1a568f5ce15c99d8cb6c61050c619bbffea45

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 244.7 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5

File hashes

Hashes for cpprb-9.2.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 bc7399b8030ad9f91b5a3673478a07a45074c3f122b2a0dba67b524dcc1dd753
MD5 5cc9a15d994a710ee482cf39e57a23ac
BLAKE2b-256 26a47380b7173f882288b32bbe49f68d9d9763eb9c2b7be4f18b58847399001f

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 5aa40e325059474d160df603126b81c307fe7bc5125407042ffbfc060148dbb5
MD5 d952a097064c3b1b59cdfa005434b6e5
BLAKE2b-256 62136bc2a4b8b47f8bf8e1ce21fd4bd25f54046ae6002b821cada680cbe82cef

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 c0ac4bd423ef455a15ecd8104be33f116d572e4195d049333cc615b4dfab52a1
MD5 28fd287decc393c6b0b92d6f6cda2d4e
BLAKE2b-256 05afb61567a45f32d63c10eef666741ecac15ee00e617b3be509dba882e3fe1d

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 236.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 de30768d3238ce1cd55ad7abb885682b251a85e806f24e150808bb2f22ad33be
MD5 f71d980a28ac30ed45ab6784fee9b54d
BLAKE2b-256 a020b54f83c179c625830834323c337aae8a80aa9dfa54969a1094c99536fa27

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 d4ee4757295c145226d052ffeaa85e5dfcbc33daf139b49268de16a8c452704e
MD5 6e6140991b6c0fa6c4b1a94a3c312b19
BLAKE2b-256 1a96b3740196600ba3ed6a80c4c636572efc62e7db9d2362bedde2b3a11acb40

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 0b92c3c51d8c0caf5ebfaea41ddba0bacf2d225e0e6e871d6ab1be5188f45d93
MD5 10001cfdb77dea447b031a0be8711032
BLAKE2b-256 0359b720df42f9fbe27721f22ccaf76122394b9877b5f860fd6a1d6303ef6d15

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 236.8 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.6.8

File hashes

Hashes for cpprb-9.2.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 b120805b19db8b00ba1bdee0e83eb1353391656046f8c14f827737b1ff00dddd
MD5 69033a2d41246886181541c531747d6e
BLAKE2b-256 8ec5547c12cb8e469342da2df836a035d42adad51f6c0577eb81535bb390e402

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 8300b16b7f134b379678193b0c3695986cb96829ef294d44c6bdcc73f2456b11
MD5 1afe9238b0a12be48027fc12fa966bf3
BLAKE2b-256 55cdf4e0aab35e5c5296125dc5c7df5e2869c266c6cbadd2fb52dfc60c3b6ea7

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a99cf0c4c95db729291d22a88bd6e0cc501bc8b1e719a4b694e2a8ac779113f7
MD5 8a02a7d8156e8fe6844ae9f46c05051f
BLAKE2b-256 8c0236e5a8a10654896d4fffb09083d391f2256db75c09d6621db096dece27f4

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp35-cp35m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp35-cp35m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.5m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp35-cp35m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 f13407e4562eaf47d46118cebff2bf137e6df94f3b1100eae6b4d1bb0e221168
MD5 5168cbc4c3bd43dadea9d7b63b4d17a6
BLAKE2b-256 05496954ffaaa780748f8be928ed9bf559e11fa6a2380ccd7db5891882bcbf00

See more details on using hashes here.

File details

Details for the file cpprb-9.2.1-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: cpprb-9.2.1-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.8

File hashes

Hashes for cpprb-9.2.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 7f6cf7591e97d1be0011d0ef0fa5181b57f84c59776ec0cd8842c67581c62d2f
MD5 023c64cb46737fc01db24fd3c80cce3d
BLAKE2b-256 a9a1117b8acffdde8e5c1f93ce6066f4abc3a443b0ad862400d84ca5d571a684

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page