dex-retargeting 0.5.0

Hand pose retargeting for dexterous robot hand.

Dex Retargeting

Various retargeting optimizers to translate human hand motion to robot hand motion.

This repo originates from AnyTeleop Project

Installation

pip install dex_retargeting

To run the example, you may need additional dependencies for rendering and hand pose detection.

git clone https://github.com/dexsuite/dex-retargeting
cd dex-retargeting
pip install -e ".[example]"

Changelog

v0.5.0

• Numpy Support Update: Starting from this version, dex-retargeting supports numpy >= 2.0.0. If you need to use numpy < 2.0.0, you can install an earlier version of dex-retargeting using:

  pip install "dex-retargeting<0.5.0"
  

• Mediapipe Compatibility: Although mediapipe lists numpy 1.x as a dependency, it is compatible with numpy >= 2.0.0. You can safely ignore any warnings related to this and continue using numpy 2.0.0 or higher.

• Dependency Cleanup: Removed trimesh as a dependency to simplify installation and reduce potential conflicts. The core functionality of dex-retargeting no longer requires mesh processing capabilities.

Examples

Retargeting from human hand video

This type of retargeting can be used for applications like teleoperation, e.g. AnyTeleop.

Tutorial on retargeting from human hand video

Retarget from hand object pose dataset

This type of retargeting can be used post-process human data for robot imitation, e.g. DexMV.

Tutorial on retargeting from hand-object pose dataset

FAQ and Troubleshooting

Joint Orders for Retargeting

URDF parsers, such as ROS, physical simulators, real robot driver, and this repository, may parse URDF files with different joint orders. To use dex-retargeting results with other libraries, handle joint ordering explicitly using joint names, which are unique within a URDF file.

Example: Using dex-retargeting with the SAPIEN simulator

from dex_retargeting.seq_retarget import SeqRetargeting

retargeting: SeqRetargeting
sapien_joint_names = [joint.get_name() for joint in robot.get_active_joints()]
retargeting_joint_names = retargeting.joint_names
retargeting_to_sapien = np.array([retargeting_joint_names.index(name) for name in sapien_joint_names]).astype(int)

# Use the index map to handle joint order differences
sapien_robot.set_qpos(retarget_qpos[retargeting_to_sapien])

This example retrieves joint names from the SAPIEN robot and SeqRetargeting object, creates a mapping array (retargeting_to_sapien) to map joint indices, and sets the SAPIEN robot's joint positions using the retargeted joint positions.

Citation

This repository is derived from the AnyTeleop Project and is subject to ongoing enhancements. If you utilize this work, please cite it as follows:

@inproceedings{qin2023anyteleop,
  title     = {AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System},
  author    = {Qin, Yuzhe and Yang, Wei and Huang, Binghao and Van Wyk, Karl and Su, Hao and Wang, Xiaolong and Chao, Yu-Wei and Fox, Dieter},
  booktitle = {Robotics: Science and Systems},
  year      = {2023}
}

Acknowledgments

The robot hand models in this repository are sourced directly from dex-urdf. The robot kinematics in this repo are based on pinocchio. Examples use SAPIEN for rendering and visualization.

The PositionOptimizer leverages methodologies from our earlier project, From One Hand to Multiple Hands. Additionally, the DexPilotOptimizeris crafted using insights from DexPilot.