CusRL: Customizable Reinforcement Learning

CusRL is a flexible and modular reinforcement-learning framework designed for customization. By breaking down complex algorithms into minimal components, it allows users to easily modify or integrate components instead of rebuilding the entire algorithm from scratch, making it particularly well-suited for advancing robotics learning.

Note: This project is under active development, which means the interface is unstable and breaking changes are likely to occur frequently.

Setup

CusRL requires Python 3.10 or later. It can be installed via PyPI with:

# Choose one of the following:
# 1. Minimal installation
pip install cusrl
# 2. Install with export and logging utilities
pip install cusrl[all]

or by cloning this repository and installing it with:

git clone https://github.com/chengruiz/cusrl.git
# Choose one of the following:
# 1. Minimal installation
pip install -e . --config-settings editable_mode=strict
# 2. Install with optional dependencies
pip install -e .[all] --config-settings editable_mode=strict
# 3. Install dependencies for development
pip install -e .[dev] --config-settings editable_mode=strict
pre-commit install

Quick Start

List all available experiments:

python -m cusrl list-experiments

Train a PPO agent and evaluate it:

python -m cusrl train -env MountainCar-v0 -alg ppo --logger tensorboard --seed 42
python -m cusrl play --checkpoint logs/MountainCar-v0_ppo

Or if you have IssacLab installed:

python -m cusrl train -env Isaac-Velocity-Rough-Anymal-C-v0 -alg ppo \
    --logger tensorboard --env-args="--headless"
python -m cusrl play --checkpoint logs/Isaac-Velocity-Rough-Anymal-C-v0_ppo

Try distributed training:

torchrun --nproc-per-node=2 -m cusrl train -env Isaac-Velocity-Rough-Anymal-C-v0 \
    -alg ppo --logger tensorboard --env-args="--headless"

Highlights

CusRL provides a modular and extensible framework for RL with the following key features:

• Modular Design: Components are highly decoupled, allowing for free combination and customization
• Diverse Network Architectures: Support for MLP, CNN, RNNs, Transformer and custom architectures
• Modern Training Techniques: Built-in support for distributed and mixed-precision training

Implemented Algorithms

• Action Smoothness Constraint
• Adversarial Motion Prior (AMP)
• Generalized Advantage Estimation (GAE) with distinct lambda values
• Privileged Training (Policy Distillation)
• Proximal Policy Optimization (PPO) with recurrent policy support
• Random Network Distillation (RND)
• Symmetry Augmentations:
  • Mirror Symmetry Loss
  • Symmetric Architecture
  • Symmetric Data Augmentation

Cite

If you find this framework useful for your research, please consider citing our work on legged locomotion:

• A Spatio-Temporal Hierarchical Framework for Legged Navigation Via Closed-Loop Dynamics Proxy Model, IROS 2026
• Efficient Learning of A Unified Policy For Whole-body Manipulation and Locomotion Skills, IROS 2025 Best Paper Award Finalist
• Learning Symmetric Legged Locomotion via State Distribution Symmetrization, IROS 2025
• Learning Accurate and Robust Velocity Tracking for Quadrupedal Robots, JFR 2025
• Learning Safe Locomotion for Quadrupedal Robots by Derived-Action Optimization, IROS 2024

Acknowledgement

CusRL is based on or inspired by these projects:

• Stable Baselines3: Reliable implementations of reinforcement learning algorithms
• RSL RL: Fast and simple implementation of RL algorithms
• IsaacLab: GPU-accelerated simulation environments for robot research
• robot_lab: RL extension library for robots, based on IsaacLab
• OnnxSlim: Library for performing optimizations on ONNX models