Skip to main content

A Library for Deep Reinforcement Learning

Project description

PyPI Conda Read the Docs Read the Docs Unittest codecov GitHub issues GitHub stars GitHub forks GitHub license

⚠️️ Current Status: the Tianshou master branch is currently under heavy development, moving towards more features, improved interfaces, more documentation. You can view the relevant issues in the corresponding milestone Stay tuned! (and expect breaking changes until the next major release)

Tianshou (天授) is a reinforcement learning platform based on pure PyTorch and Gymnasium. Unlike other reinforcement learning libraries, which may have complex codebases, unfriendly high-level APIs, or are not optimized for speed, Tianshou provides a high-performance, modularized framework and user-friendly interfaces for building deep reinforcement learning agents. One more aspect that sets Tianshou apart is its generality: it supports online and offline RL, multi-agent RL, and model-based algorithms.

Tianshou aims at enabling concise implementations, both for researchers and practitioners, without sacrificing flexibility.

Supported algorithms include:

Other noteworthy features:

  • Elegant framework with dual APIs:
    • Tianshou's high-level API maximizes ease of use for application development while still retaining a high degree of flexibility.
    • The fundamental procedural API provides a maximum of flexibility for algorithm development without being overly verbose.
  • State-of-the-art results in MuJoCo benchmarks for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
  • Support for vectorized environments (synchronous or asynchronous) for all algorithms (see usage)
  • Support for super-fast vectorized environments based on EnvPool for all algorithms (see usage)
  • Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (see usage)
  • Support any type of environment state/action (e.g. a dict, a self-defined class, ...) Usage
  • Support for customized training processes (see usage)
  • Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations
  • Support for multi-agent RL (see usage)
  • Support for logging based on both TensorBoard and W&B
  • Support for multi-GPU training (see usage)
  • Comprehensive documentation, PEP8 code-style checking, type checking and thorough tests

In Chinese, Tianshou means divinely ordained, being derived to the gift of being born. Tianshou is a reinforcement learning platform, and the nature of RL is not learn from humans. So taking "Tianshou" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.



Tianshou is currently hosted on PyPI and conda-forge. It requires Python >= 3.11.

For installing the most recent version of Tianshou, the best way is clone the repository and install it with poetry (which you need to install on your system first)

git clone
cd tianshou
poetry install

You can also install the dev requirements by adding --with dev or the extras for say mujoco and acceleration by envpool by adding --extras mujoco envpool

Available extras are:

  • atari (for Atari environments)
  • box2d (for Box2D environments)
  • classic_control (for classic control (discrete) environments)
  • mujoco (for MuJoCo environments)
  • mujoco-py (for legacy mujoco-py environments[^1])
  • pybullet (for pybullet environments)
  • robotics (for gymnasium-robotics environments)
  • vizdoom (for ViZDoom environments)
  • envpool (for envpool integration)
  • argparse (in order to be able to run the high level API examples)

[^1]: mujoco-py is a legacy package and is not recommended for new projects. It is only included for compatibility with older projects. Also note that there may be compatibility issues with macOS newer than Monterey.

Otherwise, you can install the latest release from PyPI (currently far behind the master) with the following command:

$ pip install tianshou

If you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:

$ conda install tianshou -c conda-forge

Alternatively to the poetry install, you can also install the latest source version through GitHub:

$ pip install git+ --upgrade

Finally, you may check the installation via your Python console as follows:

import tianshou

If no errors are reported, you have successfully installed Tianshou.


Tutorials and API documentation are hosted on

Find example scripts in the test/ and examples/ folders.


Why Tianshou?

Comprehensive Functionality

RL Platform GitHub Stars # of Alg. (1) Custom Env Batch Training RNN Support Nested Observation Backend
Baselines GitHub stars 9 :heavy_check_mark: (gym) :heavy_minus_sign: (2) :heavy_check_mark: :x: TF1
Stable-Baselines GitHub stars 11 :heavy_check_mark: (gym) :heavy_minus_sign: (2) :heavy_check_mark: :x: TF1
Stable-Baselines3 GitHub stars 7 (3) :heavy_check_mark: (gym) :heavy_minus_sign: (2) :x: :heavy_check_mark: PyTorch
Ray/RLlib GitHub stars 16 :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: TF/PyTorch
SpinningUp GitHub stars 6 :heavy_check_mark: (gym) :heavy_minus_sign: (2) :x: :x: PyTorch
Dopamine GitHub stars 7 :x: :x: :x: :x: TF/JAX
ACME GitHub stars 14 :heavy_check_mark: (dm_env) :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: TF/JAX
keras-rl GitHub stars 7 :heavy_check_mark: (gym) :x: :x: :x: Keras
rlpyt GitHub stars 11 :x: :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: PyTorch
ChainerRL GitHub stars 18 :heavy_check_mark: (gym) :heavy_check_mark: :heavy_check_mark: :x: Chainer
Sample Factory GitHub stars 1 (4) :heavy_check_mark: (gym) :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: PyTorch
Tianshou GitHub stars 20 :heavy_check_mark: (Gymnasium) :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: PyTorch

(1): access date: 2021-08-08

(2): not all algorithms support this feature

(3): TQC and QR-DQN in sb3-contrib instead of main repo

(4): super fast APPO!

High Software Engineering Standards

RL Platform Documentation Code Coverage Type Hints Last Update
Baselines :x: :x: :x: GitHub last commit
Stable-Baselines Documentation Status coverage :x: GitHub last commit
Stable-Baselines3 Documentation Status coverage report :heavy_check_mark: GitHub last commit
Ray/RLlib :heavy_minus_sign:(1) :heavy_check_mark: GitHub last commit
SpinningUp :x: :x: GitHub last commit
Dopamine :x: :x: GitHub last commit
ACME :heavy_minus_sign:(1) :heavy_check_mark: GitHub last commit
keras-rl Documentation :heavy_minus_sign:(1) :x: GitHub last commit
rlpyt Docs codecov :x: GitHub last commit
ChainerRL Documentation Status Coverage Status :x: GitHub last commit
Sample Factory :heavy_minus_sign: codecov :x: GitHub last commit
Tianshou Read the Docs codecov :heavy_check_mark: GitHub last commit

(1): it has continuous integration but the coverage rate is not available

Reproducible, High-Quality Results

Tianshou is rigorously tested. In contrast to other RL platforms, our tests include the full agent training procedure for all of the implemented algorithms. Our tests would fail once if any of the agents failed to achieve a consistent level of performance on limited epochs. Our tests thus ensure reproducibility. Check out the GitHub Actions page for more detail.

Atari and MuJoCo benchmark results can be found in the examples/atari/ and examples/mujoco/ folders respectively. Our MuJoCo results reach or exceed the level of performance of most existing benchmarks.

Policy Interface

All algorithms implement the following, highly general API:

  • __init__: initialize the policy;
  • forward: compute actions based on given observations;
  • process_buffer: process initial buffer, which is useful for some offline learning algorithms
  • process_fn: preprocess data from the replay buffer (since we have reformulated all algorithms to replay buffer-based algorithms);
  • learn: learn from a given batch of data;
  • post_process_fn: update the replay buffer from the learning process (e.g., prioritized replay buffer needs to update the weight);
  • update: the main interface for training, i.e., process_fn -> learn -> post_process_fn.

The implementation of this API suffices for a new algorithm to be applicable within Tianshou, making experimenation with new approaches particularly straightforward.

Quick Start

Tianshou provides two API levels:

  • the high-level interface, which provides ease of use for end users seeking to run deep reinforcement learning applications
  • the procedural interface, which provides a maximum of control, especially for very advanced users and developers of reinforcement learning algorithms.

In the following, let us consider an example application using the CartPole gymnasium environment. We shall apply the deep Q network (DQN) learning algorithm using both APIs.

High-Level API

To get started, we need some imports.

from tianshou.highlevel.config import SamplingConfig
from tianshou.highlevel.env import (
from tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig
from tianshou.highlevel.params.policy_params import DQNParams
from tianshou.highlevel.trainer import (

In the high-level API, the basis for an RL experiment is an ExperimentBuilder with which we can build the experiment we then seek to run. Since we want to use DQN, we use the specialization DQNExperimentBuilder. The other imports serve to provide configuration options for our experiment.

The high-level API provides largely declarative semantics, i.e. the code is almost exclusively concerned with configuration that controls what to do (rather than how to do it).

experiment = (
        EnvFactoryRegistered(task="CartPole-v1", seed=0, venv_type=VectorEnvType.DUMMY),
            watch_render=1 / 35,
            update_per_step=1 / 10,
    .with_model_factory_default(hidden_sizes=(64, 64))

The experiment builder takes three arguments:

  • the environment factory for the creation of environments. In this case, we use an existing factory implementation for gymnasium environments.
  • the experiment configuration, which controls persistence and the overall experiment flow. In this case, we have configured that we want to observe the agent's behavior after it is trained (watch=True) for a number of episodes (watch_num_episodes=100). We have disabled persistence, because we do not want to save training logs, the agent or its configuration for future use.
  • the sampling configuration, which controls fundamental training parameters, such as the total number of epochs we run the experiment for (num_epochs=10)
    and the number of environment steps each epoch shall consist of (step_per_epoch=10000). Every epoch consists of a series of data collection (rollout) steps and training steps. The parameter step_per_collect controls the amount of data that is collected in each collection step and after each collection step, we perform a training step, applying a gradient-based update based on a sample of data (batch_size=64) taken from the buffer of data that has been collected. For further details, see the documentation of SamplingConfig.

We then proceed to configure some of the parameters of the DQN algorithm itself and of the neural network model we want to use. A DQN-specific detail is the use of callbacks to configure the algorithm's epsilon parameter for exploration. We want to use random exploration during rollouts (train callback), but we don't when evaluating the agent's performance in the test environments (test callback).

Find the script in examples/discrete/ Here's a run (with the training time cut short):

Find many further applications of the high-level API in the examples/ folder; look for scripts ending with Note that most of these examples require the extra package argparse (install it by adding --extras argparse when invoking poetry).

Procedural API

Let us now consider an analogous example in the procedural API. Find the full script in examples/discrete/

First, import some relevant packages:

import gymnasium as gym
import torch
from torch.utils.tensorboard import SummaryWriter
import tianshou as ts

Define some hyper-parameters:

task = 'CartPole-v1'
lr, epoch, batch_size = 1e-3, 10, 64
train_num, test_num = 10, 100
gamma, n_step, target_freq = 0.9, 3, 320
buffer_size = 20000
eps_train, eps_test = 0.1, 0.05
step_per_epoch, step_per_collect = 10000, 10

Initialize the logger:

logger = ts.utils.TensorboardLogger(SummaryWriter('log/dqn'))
# For other loggers, see

Make environments:

# You can also try SubprocVectorEnv, which will use parallelization
train_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(train_num)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(test_num)])

Create the network as well as its optimizer:

from import Net

# Note: You can easily define other networks.
# See
env = gym.make(task, render_mode="human")
state_shape = env.observation_space.shape or env.observation_space.n
action_shape = env.action_space.shape or env.action_space.n
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])
optim = torch.optim.Adam(net.parameters(), lr=lr)

Set up the policy and collectors:

policy = ts.policy.DQNPolicy(
train_collector =, train_envs,, train_num), exploration_noise=True)
test_collector =, test_envs, exploration_noise=True)  # because DQN uses epsilon-greedy method

Let's train it:

result = ts.trainer.OffpolicyTrainer(
    update_per_step=1 / step_per_collect,
    train_fn=lambda epoch, env_step: policy.set_eps(eps_train),
    test_fn=lambda epoch, env_step: policy.set_eps(eps_test),
    stop_fn=lambda mean_rewards: mean_rewards >= env.spec.reward_threshold,
print(f"Finished training in {result.timing.total_time} seconds")

Save/load the trained policy (it's exactly the same as loading a torch.nn.module):, 'dqn.pth')

Watch the agent with 35 FPS:

collector =, env, exploration_noise=True)
collector.collect(n_episode=1, render=1 / 35)

Inspect the data saved in TensorBoard:

$ tensorboard --logdir log/dqn

Please read the documentation for advanced usage.


Tianshou is still under development. Further algorithms and features are continuously being added, and we always welcome contributions to help make Tianshou better. If you would like to contribute, please check out this link.

Citing Tianshou

If you find Tianshou useful, please cite it in your publications.

  author  = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},
  title   = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {267},
  pages   = {1--6},
  url     = {}


Tianshou is supported by appliedAI Institute for Europe, who is committed to providing long-term support and development.

Tianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branch priv for more detail. Many thanks to Haosheng Zou's pioneering work for Tianshou before version 0.1.1.

We would like to thank TSAIL and Institute for Artificial Intelligence, Tsinghua University for providing such an excellent AI research platform.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tianshou-1.0.0.tar.gz (178.4 kB view hashes)

Uploaded Source

Built Distribution

tianshou-1.0.0-py3-none-any.whl (233.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page