sapien 3.0.3

['SAPIEN: A SimulAted Parted based Interactive ENvironment']

SAPIEN

SAPIEN is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. It enables various robotic vision and interaction tasks that require detailed part-level understanding. SAPIEN is a collaborative effort between researchers at UCSD, Stanford and SFU. The dataset is a continuation of ShapeNet and PartNet.

Getting Started

SAPIEN is distributed via PyPI. Installation is just

pip install sapien

It requires Linux with NVIDIA, AMD, or Intel GPU to run. Verify installation with

python -m sapien.exapmle.hello_world

Next, follow our tutorial at: https://sapien-sim.github.io/docs/.

Offscreen rendering on a server

To use SAPIEN on a GPU server without display, the only system dependencies required are libegl1 and libxext6. If using NVIDIA docker environment, enable graphics, utility, and compute by setting the environment variable in the Dockerfile.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute

Virtual desktop on a server

To use SAPIEN on a GPU server with virtual display, additionally install xvfb, x11vnc, and any window manager such as fluxbox or xfce. Add display capabilities for NVIDIA docker.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute,display

Assuming fluxbox, start a VNC server by

x11vnc -create -env FD_PROG=/usr/bin/fluxbox  -env X11VNC_FINDDISPLAY_ALWAYS_FAILS=1 -env X11VNC_CREATE_GEOM=${99:-1920x1080x16} -gone 'pkill Xvfb' -nopw
# Note: you should use a strong password and/or only allow local access

Now you can connect to the server at port 5900. SAPIEN should be fully functional, test with

python -m sapien.exapmle.hello_world

Change Log

3.0

• Major API & infrastructure overhaul
  • Refactor SAPIEN core to use entity-component system
    • Actor from SAPIEN 2 now becomes Entity in SAPIEN 3
    • Functionalities of an Actor now becomes components attached to an entity.
    • See updated documentation for usage.
    • Actor builder and articulation builder APIs are mostly unchanged, except the color and material parameters for visual shapes are unified to material.
  • Support for PhysX 5 GPU simulation. See ManiSkill for how it is used to build robot learning environments.

2.2

• Rename VulkanRenderer to SapienRenderer (VulkanRenderer is still an alias)
• Support ray tracing in SapienRenderer
• Deprecate KuafuRenderer, use the rt shader in SapienRenderer instead
• GPU-accelerated stereo depth sensor simulation
• Render server
• Python 3.11
• bug fixes
  • Fix inverse kinematics default active joint mask (now defaults to all 1s)
  • Fix incorrectly exported memory in Vulkan-Cuda interop
  • Fix joint get_global_pose

2.1

• Python 3.10
• Bug fixes
  • crash when not using renderer
  • joint force limit (was impulse limit)
  • incorrect inertia computation in scaled URDF
  • incorrect point-light shadow
  • incorrect collision when loaded from dae
• Utility improvements
  • set_material
  • active light
  • flat shading
  • dynamic point rendering
  • envmap generation
  • multi-thread envs

2.1

• Refactor light system
  • Remove light functions on scene.renderer_scene
• Refactor camera system
  • Cameras no longer require mounts
  • Camera can change its mount and mounted pose by camera.set_parent and camera.set_local_pose.
  • When camera is not mounted, setting local pose is setting its global pose.
  • Add functions scene.add_camera and scene.remove_camera
  • add_mounted_camera can be replaced with add_camera followed by camera.set_parent and camera.set_local_pose. add_mounted_camera is still provided but fovx should not longer be provided.
  • Remove functions related to mount, including find_camera_by_mount.
  • Cameras now support full camera parameters through camera.near, camera.far, camera.set_fovx, camera.set_fovy, camera.set_focal_lengths, camera.set_principal_point, camera.skew, and the all-in-one method camera.set_perspective_parameters.
• Refactor render shape system
  • Originally, after actor.get_visual_bodies() and visual_body.get_render_shapes(), users typically do shape.scale and shape.pose. These are no longer valid. It is required to check visual_body.type. When type is mesh, shape.scale is replaced with visual_body.scale and shape.pose is replaced by visual_body.local_pose. These changes are made to match add_visual_shape functions when building the actor.

pre2.0

• Shader change: 4th component in default camera shader now gives the 0-1 depth value.
• Add "critical" and "off" log levels.
• Add support for pointcloud and line rendering (for visualizing camera and point cloud)
• Performance: the same shader only compile once per process
• Bug fix
  • Articulation setDriveTarget was now correctly reversed for prismatic joint (joint setDriveTarget is not affected)
  • Fix kinematic articulation loader

1 to 2 migration

• replace scene.renderer_scene.add_xxx_light with scene.add_xxx_light
• replace scene.remove_mounted_camera with scene.remove_camera
• optionally, remove fovx from scene.add_mounted_camera.

1.1

• Support nonconvex static/kinematic collision shape
• Add warning for small mass/inertia
• Introduce Entity as the base class of Actors
• Add Light classes inherited from entity, allowing manipulate light objects in sapien scene
• Updates to the viewer
  • rename actor to entity when appropriate
• Partial support the material tag in URDF loader (primitive shape, single color)
• Bug fixes for the renderer
• Support inner and outer FOV for spotlight

1.0

• Replace the old Vulkan based renderer completely
  • See sapien.core.renderer for details
• Expose GUI functionalities to Python
• Reimplement Vulkan viewer in Python
• Expose PhysX shape wrapper to Python. For example,
  • Collision shapes can be retrieved through actor.get_collision_shapes
  • Collision groups on a shape can be set by CollisionShape.set_collision_groups
  • Shapes are now also available in Contact.
• API changes
  • Render material creation is now renderer.create_material()
  • in actor builder: add_xxx_shape is replaced with add_xxx_collision.
  • move light functions from scene to scene.renderer_scene
• Add centrifugal and Coriolis force.
• Change default physical parameters for better stability.

Website and Documentation

SAPIEN Website: https://sapien.ucsd.edu/. SAPIEN Documentation: https://sapien-sim.github.io/docs/.

Build from source

Before build

Make sure all submodules are initialized git submodule update --init --recursive.

Build with Docker

To build SAPIEN, simply run ./scripts/docker_build_wheels.sh. It is not recommended to build outside of our provided docker.

For reference, the Dockerfile is provided here. Note that PhysX needs to be compiled with clang-9 into static libraries before building the Docker image.

Build without Docker

It can be tricky to setup all dependencies outside of a Docker environment. You need to install all dependencies according to the Docker environment. If all dependencies set up correctly, run python setup.py bdist_wheel to build the wheel.

Cite SAPIEN

If you use SAPIEN and its assets, please cite the following works:

@InProceedings{Xiang_2020_SAPIEN,
author = {Xiang, Fanbo and Qin, Yuzhe and Mo, Kaichun and Xia, Yikuan and Zhu, Hao and Liu, Fangchen and Liu, Minghua and Jiang, Hanxiao and Yuan, Yifu and Wang, He and Yi, Li and Chang, Angel X. and Guibas, Leonidas J. and Su, Hao},
title = {{SAPIEN}: A SimulAted Part-based Interactive ENvironment},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}}

@InProceedings{Mo_2019_CVPR,
author = {Mo, Kaichun and Zhu, Shilin and Chang, Angel X. and Yi, Li and Tripathi, Subarna and Guibas, Leonidas J. and Su, Hao},
title = {{PartNet}: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level {3D} Object Understanding},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}

@article{chang2015shapenet,
title={Shapenet: An information-rich 3d model repository},
author={Chang, Angel X and Funkhouser, Thomas and Guibas, Leonidas and Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and Savva, Manolis and Song, Shuran and Su, Hao and others},
journal={arXiv preprint arXiv:1512.03012},
year={2015}
}

If you use SAPIEN Realistic Depth generated by SAPIEN's simulated depth sensor, please cite the following work:

@ARTICLE{10027470,
  author={Zhang, Xiaoshuai and Chen, Rui and Li, Ang and Xiang, Fanbo and Qin, Yuzhe and Gu, Jiayuan and Ling, Zhan and Liu, Minghua and Zeng, Peiyu and Han, Songfang and Huang, Zhiao and Mu, Tongzhou and Xu, Jing and Su, Hao},
  journal={IEEE Transactions on Robotics}, 
  title={Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation}, 
  year={2023},
  volume={},
  number={},
  pages={1-19},
  doi={10.1109/TRO.2023.3235591}}