videosdk-teleop

Drive a robot arm over the internet, and know exactly what the operator was looking at when they moved it.

A WebRTC transport for robot teleoperation, with frame-accurate observation/action correlation, a safety envelope that cannot be bypassed, and a recorder that turns every session into training data.

Install

pip install videosdk-teleop               # transport + cameras, ready to run
pip install "videosdk-teleop[lerobot]"    # + the lerobot adapters
pip install "videosdk-teleop[ros2]"       # + the ROS 2 nodes

Python 3.10-3.12. The first line is enough to teleoperate: the WebRTC transport and the camera stack are core dependencies, not extras.

Your robot, in three methods

Everything else is inherited: lease negotiation, clamping, the watchdog, the e-stop latch, correlation, recording.

from videosdk_teleop import Follower, SafetyConfig, single_group_descriptor

class MyArm(Follower):
    def descriptor(self):            return single_group_descriptor(...)
    def read_joints(self):           return {"shoulder": 12.4, "elbow": -3.1}
    def write_joints(self, joints):  ...        # RAISE on failure

MyArm("abcd-efgh-ijkl", safety=SafetyConfig(slew=12.0)).run(hz=50)

Already on lerobot? Wrap the device instead of subclassing:

from videosdk_teleop.adapters.lerobot import (SO101FollowerAdapter,
                                              SO101LeaderAdapter)

follower = SO101FollowerAdapter("abcd-efgh-ijkl", robot=arm, token=TOKEN)

On ROS 2, teleop_bridge_node does the same against your driver topics.

Meeting ids come from the VideoSDK API, or videosdk_teleop.rooms.create_room().

Run it locally, no robot needed

Runnable examples live in videosdk-teleops-examples:

git clone https://github.com/videosdk-live/videosdk-teleops-examples
cd videosdk-teleops-examples

python quickstart/remote_teleop.py --ticks 100   # full chain, in-process
python quickstart/my_arm.py                      # port your own arm
python quickstart/protocol_demo.py               # the wire format, live

remote_teleop.py runs a real follower against a real leader: real gates, real clamps, real observation log. Only the network is swapped for a loopback.

You own the loop

leader.tick()                 # one control period, never blocks
leader.run(hz=50)             # or let it pace the loop for you

for _ in leader.ticks(hz=50):
    obs = leader.observation()          # joints + pixels, same instant
    print(obs.observation_id, obs.joints, obs.images)

    frame = leader.peek_synced()        # watch only: does not arm the echo

Safety

SafetyConfig(
    slew=12.0,                          # max change per joint per tick
    watchdog_timeout_s=0.5,             # silence for this long -> failsafe
    max_staleness_ms=100.0,             # older commands are dropped
    require_deadman=True,
    on_starvation=FailsafeAction.HOLD,
    resume_requires_reclaim=False)

follower.estop()          # unauthenticated by design: anyone can stop an arm
follower.clear_estop()    # latched: only a human on the follower host resumes

Every command is checked before it reaches the motors: who sent it, whether it arrived too late, and how far it moves the arm. If the operator stops sending, the arm holds where it is.

Knowing what the operator saw

Every action is tagged with the exact camera frame the operator was looking at when they sent it. The tag travels with the command, so the two machines never need their clocks in sync.

Recording

follower.start_recording("./sessions")
follower.start_episode("pick up the cube")
...                                          # your loop runs
follower.end_episode(success=True)
follower.stop_recording()

On the follower this captures pixels before an encoder or a network touched them. Episodes open and close on the deadman by default.

VideoSDK can also record the session in the cloud:

follower = MyArm("abcd-efgh-ijkl", cloud_recording=True)

follower.cloud_recording_state    # idle / requested / starting / ready / failed

If the cloud recorder never comes up, recording carries on locally and the state says so.

Watching a session

from videosdk_teleop import attach

attach(follower, host="0.0.0.0", port=8080)   # live page + /snapshot.json

On ROS 2

One node, teleop_bridge_node, one role per host. It never talks to your motors directly: it reads your driver's joint-state topic and writes to your driver's joint-command topic, exactly like any other ROS node.

ros2 launch videosdk_teleop_ros bridge.launch.py \
    role:=follower_side robot_id:=my_arm cameras:=cam_a,cam_b labels:=wrist,front

ros2 service call /videosdk_bridge/enable std_srvs/srv/SetBool "{data: true}"

The bridge sits idle until enable is called, so launching it moves nothing.

Install into the same Python that ROS 2 uses, not whatever python3 your shell resolves to:

/usr/bin/python3 -m pip install --user videosdk-teleop

Documentation

Full documentation at docs.videosdk.live.

Apache-2.0 · docs.videosdk.live · examples