Overview
cpprb is a python (CPython) module providing replay buffer classes for reinforcement learning.
Major target users are researchers and library developers.
You can build your own reinforcement learning algorithms together with your favorite deep learning library (e.g. TensorFlow, PyTorch).
cpprb forcuses speed, flexibility, and memory efficiency.
By utilizing Cython, complicated calculations (e.g. segment tree for prioritized experience replay) are offloaded onto C++. (The name cpprb comes from "C++ Replay Buffer".)
In terms of API, initially cpprb referred to OpenAI Baselines' implementation. In the current version, cpprb has much more flexibility. Any NumPy compatible types of any numbers of values can be stored (as long as memory capacity is sufficient). For example, you can store the next action and the next next observation, too.
Installation
cpprb requires following softwares before installation.
- C++17 compiler (for installation from source)
- GCC (maybe 7.2 and newer)
- Visual Studio (2017 Enterprise is fine)
- Python 3
- pip
Cuurently, clang, which is a default Xcode C/C++ compiler at Apple macOS, cannot compile cpprb.
If you are macOS user, you need to install GCC and set environment values
of CC and CXX to g++, or just use virtual environment (e.g. Docker).
Step by step installation is described here.
Additionally, here are user's good feedbacks for installation at macOS and Ubuntu. (Thanks!)
Install from PyPI (Recommended)
The following command installs cpprb together with other dependancies.
pip install cpprb
Depending on your environment, you might need sudo or --user flag
for installation.
On supported platflorms (Linux x86-64 and Windows amd64), binary packages are hosted on PyPI can be used, so that you don't need C++ compiler.
If you have trouble to install from binary, you can fall back to
source installation to passk --no-binary option to the above pip command.
Currently, no other platforms, such as macOS, and 32bit or arm-architectured Linux and Windows, cannot install from binary, and need to compile by yourself. Please be patient, we will plan to support wider platforms in future.
Install from source code
First, download source code manually or clone the repository;
git clone https://gitlab.com/ymd_h/cpprb.git
Then you can install same way;
cd cpprb
pip install .
For this installation, you need to convert extended Python (.pyx) to C++ (.cpp) during installation, it takes longer time than installation from PyPI.
Usage
Here is a simple example for storing standard environment (aka. "obs", "act", "rew", "nextobs", and "done").
from cpprb import ReplayBuffer
buffer_size = 256
obs_shape = 3
act_dim = 1
rb = ReplayBuffer(buffer_size,
env_dict ={"obs": {"shape": obs_shape},
"act": {"shape": act_dim},
"rew": {},
"next_obs": {"shape": obs_shape},
"done": {}})
obs = np.ones(shape=(obs_shape))
act = np.ones(shape=(act_dim))
rew = 0
next_obs = np.ones(shape=(obs_shape))
done = 0
for i in range(500):
rb.add(obs=obs,act=act,rew=rew,next_obs=next_obs,done=done)
batch_size = 32
sample = rb.sample(batch_size)
# sample is a dictionary whose keys are 'obs', 'act', 'rew', 'next_obs', and 'done'
Flexible environment values are defined by env_dict when buffer creation.
Since stored values have flexible name, you have to pass to
ReplayBuffer.add member by keyword.
Features
cpprb provides buffer classes for building following algorithms.
| Algorithms | cpprb class | Paper |
|---|---|---|
| Experience Replay | `ReplayBuffer` | [L. J. Lin](https://link.springer.com/article/10.1007/BF00992699) |
| Prioritized Experience Replay | `PrioritizedReplayBuffer` | [T. Schaul et. al.](https://arxiv.org/abs/1511.05952) |
| Multi-step Learning | `ReplayBuffer`, `PrioritizedReplayBuffer` |
cpprb features and its usage are described at following pages:
- Flexible Environment
- Multi-step add
- Prioritized Experience Replay
- Nstep Experience Replay
- Memory Compression
Contributing to cpprb
Any contribution are very welcome!
Making Community Larger
Bigger commumity makes development more active and improve cpprb.
- Star this repository (and/or GitHub Mirror)
- Publish your code using cpprb
- Share this repository to your friend and/or followers.
Report Issue
When you have any problems or requests, you can check issues on GitLab.com. If you still cannot find any information, you can open your own issue.
Merge Request (Pull Request)
cpprb follows local rules:
- Branch Name
- "HotFix***" for bug fix
- "Feature***" for new feature implementation
- docstring
- Must for external API
- Numpy Style
- Unit Test
- Put test code under "test/" directory
- Can test by
python -m unittest <Your Test Code>command - Continuous Integration on GitLab CI configured by
.gitlab-ci.yaml
- Open an issue and associate it to Merge Request
Step by step instruction for beginners is described at here.
Links
cpprb sites
cpprb users' repositories
- keiohta/TF2RL: TensorFlow2.0 Reinforcement Learning
Lisence
cpprb is available under MIT lisence.
1 maintainer
Metadata
1 maintainer
Release files for cpprb 8.4.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| cpprb-8.4.7.tar.gz | 309.7 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| cpprb-8.4.7-cp38-cp38-win_amd64.whl | CPython 3.8 | CPython 3.8 | Windows x86-64 | Details |
| cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl | CPython 3.8 | CPython 3.8 | Linux glibc 2.12+ x86-64 | Details |
| cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl | CPython 3.8 | CPython 3.8 | Linux glibc 2.5+ x86-64 | Details |
| cpprb-8.4.7-cp37-cp37m-win_amd64.whl | CPython 3.7 | CPython 3.7 pymalloc | Windows x86-64 | Details |
| cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl | CPython 3.7 | CPython 3.7 pymalloc | Linux glibc 2.12+ x86-64 | Details |
| cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl | CPython 3.7 | CPython 3.7 pymalloc | Linux glibc 2.5+ x86-64 | Details |
| cpprb-8.4.7-cp36-cp36m-win_amd64.whl | CPython 3.6 | CPython 3.6 pymalloc | Windows x86-64 | Details |
| cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl | CPython 3.6 | CPython 3.6 pymalloc | Linux glibc 2.12+ x86-64 | Details |
| cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl | CPython 3.6 | CPython 3.6 pymalloc | Linux glibc 2.5+ x86-64 | Details |
| cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl | CPython 3.5 | CPython 3.5 pymalloc | Linux glibc 2.12+ x86-64 | Details |
| cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl | CPython 3.5 | CPython 3.5 pymalloc | Linux glibc 2.5+ x86-64 | Details |
Total release size: 13.0 MB
Release files / cpprb-8.4.7.tar.gz
| Download URL | cpprb-8.4.7.tar.gz |
|---|---|
| Size | 309.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45b4798b974113983328e60d3187a3b80b66520b435f67cfb048ea04ec447b85
|
|
BLAKE2b-256 checksum How to use checksums |
60d2d6b362596cbe6409cc73795536ac399d53d4d40652dacc5f8fcea60e0c83
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp38-cp38-win_amd64.whl
| Download URL | cpprb-8.4.7-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 245.0 kB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
1f604ed50c78ed994787172f823d723ade6c1b7d56bc4a2df3f896327352a5fd
|
|
BLAKE2b-256 checksum How to use checksums |
65ab20de796fa73df045339c8353d63423361eaf6b3e7229b86d0820807989bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.8.2
|
Release files / cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl
| Download URL | cpprb-8.4.7-cp38-cp38-manylinux2010_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.8 Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
e83cdd4c2294f7729f505aa027146f3306f8cc9c256fb8bc0401a5486c94ff89
|
|
BLAKE2b-256 checksum How to use checksums |
34f1e68803543da2143abcd5acae09c407ea1c414193cd90e839747080af5aa1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl
| Download URL | cpprb-8.4.7-cp38-cp38-manylinux1_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.8 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
76a60dfe202f4fadd7d835f59bfd7ea9b3c75c0b758235af1fe69312592de88a
|
|
BLAKE2b-256 checksum How to use checksums |
7000242b02f6729f12e376673c738481dfb0d2eb4a070e62c974afaae2f5ec3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp37-cp37m-win_amd64.whl
| Download URL | cpprb-8.4.7-cp37-cp37m-win_amd64.whl |
|---|---|
| Size | 237.4 kB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
98847ae4651db04bcfffd4a1b216e35c9a9975befe86d47c5cde810e054adb93
|
|
BLAKE2b-256 checksum How to use checksums |
a3a4d36e027590809291a17d4690a0d733a3fdfa0978c0fe832ab5cca2e96d2b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl
| Download URL | cpprb-8.4.7-cp37-cp37m-manylinux2010_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
e5010ad443d102286fa3454effa3c55106e7d1722e1848efcb1a1e3b6199b258
|
|
BLAKE2b-256 checksum How to use checksums |
0f4581dc7644443debf2317e941492614a2aaef4741bd82d05a67f4d63f20e98
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl
| Download URL | cpprb-8.4.7-cp37-cp37m-manylinux1_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
3b7628431b6b9f03d33d8b1851fc0099dcbfb3789952a99d971731b3b71100df
|
|
BLAKE2b-256 checksum How to use checksums |
89adba8485368e248bca4e618a6acd76a4759c1beeff7bfa666201832cdc6f70
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp36-cp36m-win_amd64.whl
| Download URL | cpprb-8.4.7-cp36-cp36m-win_amd64.whl |
|---|---|
| Size | 237.4 kB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5818d5674f030812c8925449974931f6a715ebdf90be44f377ca23f911354192
|
|
BLAKE2b-256 checksum How to use checksums |
5291ff6288d5f1e022b25e5283a5149ccba62493848ac04e3e726f3a42e5c8ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.8
|
Release files / cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl
| Download URL | cpprb-8.4.7-cp36-cp36m-manylinux2010_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
6d093fc1edd02c165f5aee7ab2c66dec9887a04b10b664ebcbe429df05615c50
|
|
BLAKE2b-256 checksum How to use checksums |
7cf7501de4d7eeb93eb405847506d61e0c6c0f0f865fa97257b3f2a315508cff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl
| Download URL | cpprb-8.4.7-cp36-cp36m-manylinux1_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.6 CPython 3.6 pymalloc Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
524ebe6dd60b52ef37044c3b5883dd545bab50ed5c50cc483e0bfc9ea80278b2
|
|
BLAKE2b-256 checksum How to use checksums |
a37d2759aa9283f8fa7c43b67c20d10177d742fd2e76701efa89e2aa94707834
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl
| Download URL | cpprb-8.4.7-cp35-cp35m-manylinux2010_x86_64.whl |
|---|---|
| Size | 1.4 MB |
| Tags | CPython 3.5 CPython 3.5 pymalloc Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
f779e985400b4fc48e6f26df6057c5f96fc54de3b5f587d2ef430d228a3c4193
|
|
BLAKE2b-256 checksum How to use checksums |
db919e7766b6717bd829fc4ddca71f2291bdd6680d6408841e323052fbc11a62
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
Release files / cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl
| Download URL | cpprb-8.4.7-cp35-cp35m-manylinux1_x86_64.whl |
|---|---|
| Size | 1.4 MB |
| Tags | CPython 3.5 CPython 3.5 pymalloc Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
d77c693a24e8d0af1e1e480e69447ad333e32a3481b72494da99628d086245c9
|
|
BLAKE2b-256 checksum How to use checksums |
8e9707e7a01b44dde555f236c93c05756d9b36120404cacceab9397ec1db5827
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7
|
