RealRobot

Real robot interface and utilities for asynchronous real-time deployment.
Previously developed to connect with ManiSkill2.

Runs camera capturing and visualization as separate processes to make it closer to using ROS.

Installation

From PyPI

python3 -m pip install -U real_robot

For different functionalities, you can install the optional dependencies as follows:

• real_robot: SharedObject support only, please only use submodule real_robot.utils.multiprocessing.
• real_robot[vis]: support for most visualization, use submodule real_robot.utils.visualization.
• real_robot[vis_all]: include support for URDF.
• real_robot[all]: all optional dependencies (e.g., pyrealsense2, sapien, etc.).

If you need to install the optional xarm dependency, you must install from GitHub repo

python3 -m pip install -U real_robot[all,xarm]@git+https://github.com/KolinGuo/RealRobot.git

Run unittests with

python3 -m pip install -U pytest
python3 -m pytest [--collect-only]

Calibrated camera poses are stored in real_robot/assets/hec_camera_poses/ and loaded in real_robot/sensors/camera.py.

SharedObject Notes

• When using real_robot.utils.multiprocessing.SharedObject, you might encounter the following UserWarning when Ctrl-C your program:

  /usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 10 leaked shared_memory objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '
  

  This is because of the undefined order of processes (main and child processes) shutdown. You can safely ignore it as long as there's no any shared memory object that's created by you under /dev/shm left.

• If you see warning log messages similar to the following

  Implicitly overwriting data of <SharedObject: name=start_rs_front_camera, data_type=<class 'bool'>, nbytes=10>
  

  this can mean one of the following:

  • the shared_memory objects created by previous processes are not properly cleaned up.
    Run

    find /dev/shm -user $USER -exec rm -v {} \;
    

    to clean them all up.
  • there's a bug in your usage of SharedObject.
    By design, SharedObject cannot mutate its data type (and data size) after created.
    So you should first create a new SharedObject in process A with SharedObject("test", data=np.ones(10)) to initialize it.
    Then in process B, you can create it with SharedObject("test") and then use the .fetch()/.assign() methods.
    The warning will appear if the order is reversed.
    To control the order, you can use the helper methods start_and_wait_for_process() and signal_process_ready() under real_robot.utils.multiprocessing to synchronize the processes.

Helpful Tools

• Capture Color/Depth/IR images from RS camera, do

  python3 -m real_robot.tools.rs_capture
  

• Get detailed device information of connected RealSense device (e.g., supported stream configs, sensor intrinsics, extrinsics between sensors), do

  python3 -m real_robot.tools.enumerate_rs_devices > info.log
  

• Convert recorded ROS bag file and save rgb & depth images into .npz, do

  python3 -m real_robot.tools.convert_rosbag_to_frames <bag_path>
  

TODO

• Add type support for SharedObject
  • complex
  • bytearray
  • tuple
  • list
  • set
  • dict
    • [Preliminary] Support only 1 np.ndarray as values
• Add string length? A 8-bytes length of strings is 2**(8*8-1)-1 = 8 exa bytes, which should be enough.
• Ensure compound data types works with recursive compound structure
• Add prefixes for all SharedMemory created.
• Proper clean up zombie SharedMemory before or after multiprocessing code

Notes

To fix an unindexed rosbag recorded from RSDevice, do

apt install -y python3-rosbag
rosbag reindex <bag_path>

Known Issue

• When controlling xArm7 with motion_mode="cartesian_online" and wait=True, sometimes the robot will be stuck indefinitely in wait_move() due to get_state() returns wrong state (i.e., returns state=1 thinking it's still in motion). Simple solution can be just to control briefly via UFACTORY Studio to get it unstuck.

Changelog