cusrl 1.0.0

Customizable and modular RL algorithms implemented in PyTorch

CusRL: Customizable Reinforcement Learning

CusRL is a flexible and modular reinforcement learning framework that emphasizes customization. Its clean and decoupled implementation allows researchers to easily integrate new components, which is particularly useful for advancements in robotics learning.

Note: This project is under active development, which means the interface is unstable and breaking changes are likely to occur frequently.

Installation

Requires Python >= 3.10.

git clone https://github.com/chengruiz/cusrl.git
# Minimal installation
pip install -e . --config-settings editable_mode=strict
# Install with all optional dependencies
pip install -e .[all] --config-settings editable_mode=strict
# For development, install pre-commit (assuming you have pre-commit installed)
pre-commit install

Quick Start

Try to train a PPO agent with CusRL and evaluate it:

python -m cusrl.launch.train -env MountainCar-v0 -alg ppo --logger tensorboard --seed 42
python -m cusrl.launch.play --checkpoint logs/MountainCar-v0:ppo

Or if you have IssacLab installed:

python -m cusrl.launch.train -env Isaac-Velocity-Rough-Anymal-C-v0 -alg ppo \
    --logger tensorboard --environment-args="--headless"
python -m cusrl.launch.play --checkpoint logs/Isaac-Velocity-Rough-Anymal-C-v0:ppo

Try distributed training:

torchrun --nproc-per-node=2 -m cusrl.launch.train -env Isaac-Velocity-Rough-Anymal-C-v0 \
    -alg ppo --logger tensorboard --environment-args="--headless"

Highlights

CusRL provides a modular and extensible framework for RL with the following key features:

• Modular Design: Components are highly decoupled, allowing for easy customization and extension
• Diverse Network Architectures: Support for MLP, CNN, RNNs, Transformer and custom architectures
• Modern Training Techniques: Built-in support for distributed and mixed-precision training

CusRL is designed for researchers and practitioners who need a clean, extensible framework for implementing and experimenting with reinforcement learning algorithms. The architecture emphasizes clean separation of concerns, allowing users to modify specific components without disrupting the rest of the system.

Implemented Algorithms

• Proximal Policy Optimization (PPO) with recurrent policy support
• Generalized Advantage Estimation (GAE) with distinct lambda values
• Preserving Outputs Precisely, while Adaptively Rescaling Targets (Pop-Art)
• Random Network Distillation (RND)
• Symmetry Augmentations: Symmetry Loss, Symmetric Architecture, Symmetric Data Augmentation

Cite

If you find this framework useful for your research, please consider citing our work on legged locomotion:

• Efficient Learning of A Unified Policy For Whole-body Manipulation and Locomotion Skills, Accepted by IROS 2025
• Learning Accurate and Robust Velocity Tracking for Quadrupedal Robots, Accepted by JFR
• Learning Safe Locomotion for Quadrupedal Robots by Derived-Action Optimization, Published in IROS 2024