Rewarped

Rewarped 🌀 is a platform for reinforcement learning in parallel differentiable multiphysics simulation, built on NVIDIA/warp. Rewarped supports:

• parallel environments: to train RL agents at scale.
• differentiable simulation: to compute batched analytic gradients for optimization.
• multiphysics: (CRBA, FEM, MPM, XPBD) physics solvers and coupling to support interaction between rigid bodies, articulations, and various deformables.

We use Rewarped to re-implement a variety of RL tasks from prior works, and demonstrate that first-order RL algorithms (which use differentiable simulation to compute first-order analytic gradients) can be scaled to a range of challenging manipulation and locomotion tasks that involve interaction between rigid bodies, articulations, and deformables.

For control and reinforcement learning algorithms, see etaoxing/mineral.

Setup

We have tested on the following environment: RTX 4090, Ubuntu 22.04, CUDA 12.5, Python 3.10, PyTorch 2.

conda create -n rewarped python=3.10
conda activate rewarped

pip install torch torchvision
pip install gym==0.23.1
pip install rewarped

# --- Example: trajectory optimization
python -m rewarped.envs.warp_examples.bounce --num_envs 4
# will create a `.usd` file in `outputs/`
# use MacOS Preview or alternatives to view

# --- Example: (first-order) reinforcement learning
pip install git+https://github.com/etaoxing/mineral

python -m mineral.scripts.run \
  task=Rewarped agent=DFlexAntSAPO task.env.env_name=Ant task.env.env_suite=dflex \
  logdir="workdir/RewarpedAnt4M-SAPO/$(date +%Y%m%d-%H%M%S.%2N)" \
  agent.shac.max_epochs=2000 agent.shac.max_agent_steps=4.1e6 \
  agent.network.actor_kwargs.mlp_kwargs.units=\[128,64,32\] \
  agent.network.critic_kwargs.mlp_kwargs.units=\[64,64\] \
  run=train_eval seed=1000

See mineral/docs/rewarped.md for scripts to reproduce the original paper experiments on rewarped==1.3.0.

Usage

Tasks

dflex • AntRun	isaacgymenvs • HandReorient	plasticinelab • RollingFlat
gradsim • SoftJumper	dexdeform • HandFlip	softgym • FluidMove

Creating a new task

Copy and modify rewarped/envs/template/task.py into rewarped/envs/<suite>/<task>.py. Put assets in assets/<suite>/.

Contributing

Contributions are welcome! Please refer to CONTRIBUTING.md.

Versioning

Versions take the format X.Y.Z, where:

• X.Y matches to version X.Y.* of NVIDIA/warp.
• Z is incremented for bug fixes or new features.

Compatibility with newer or older versions of NVIDIA/warp may work, but is not guaranteed or supported. There can be some breaking changes between minor versions.

Acknowledgements

Differentiable Simulation

• NVIDIA/warp
• NVlabs/DiffRL
• MPM
  • PingchuanMa/SGA, PingchuanMa/NCLaw
  • sizhe-li/DexDeform
  • hzaskywalker/PlasticineLab

Tasks (alphabetical)

• sizhe-li/DexDeform
• NVlabs/DiffRL
• hzaskywalker/PlasticineLab
• gradsim/gradsim
• isaac-sim/IsaacGymEnvs
• leggedrobotics/legged_gym
• Xingyu-Lin/softgym
• ...

Citing

@inproceedings{xing2025stabilizing,
  title={Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation},
  author={Xing, Eliot and Luk, Vernon and Oh, Jean},
  booktitle={International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=DRiLWb8bJg}
}