videosdk-teleop

Robot teleoperation over WebRTC. A transport for lerobot, with a first-class link between what the operator saw and what the robot did — and a recording that keeps it, all the way to a LeRobot dataset.

remote operator ─► video observations ─► observation/action correlation
      ─► WebRTC teleoperation ─► safety-controlled robot
      ─► demonstration recording ─► LeRobot dataset ─► policy

Your arm stays an ordinary lerobot Robot; your leader stays an ordinary lerobot Teleoperator. SO101FollowerAdapter / SO101LeaderAdapter wrap one of each and own the network session in its place — joining the meeting, publishing joints and video, applying incoming actions, clamping them first — so any arm lerobot already supports is a few lines from teleoperated over WebRTC. The host owns the control loop (tick() / run()), not lerobot's own script, which is what lets a ROS 2 timer or a bare Python for-loop drive the same class just as well.

# on the machine holding the arm
class MyArm(Follower):
    def descriptor(self):    return single_group_descriptor(...)
    def read_joints(self):   return {...}
    def write_joints(self, joints): ...   # RAISE on failure

MyArm("meeting-id", safety=SafetyConfig(slew=12.0)).run(hz=50)

---

Install

pip install videosdk-teleop               # transport + cameras, ready to run
pip install "videosdk-teleop[lerobot]"    # + the lerobot adapters
pip install "videosdk-teleop[ros2]"       # + the ROS 2 nodes
pip install -e .                          # from a checkout

That first line is enough to run a session: it brings the WebRTC transport and the camera stack with it. The extras add a hardware bus or dataset export.

Imports stay lazy regardless, so the protocol, safety, correlation and recording layers still work on a machine where the WebRTC stack cannot load — videosdk fails to import on Python 3.14, and the transport reports itself unavailable rather than taking the rest of the SDK down with it.

Watching a session

Either side can serve a live page: camera feeds on top, stats below, one half each. The stats half is two labelled bands — network (latency, jitter, packet loss, bandwidth in/out, data in/out) and then this machine's role: leader adds actions sent, follower adds actions applied, observations and packets dropped. Glass-to-glass latency is on both. Each tile carries a 120-second line chart with axes. One screen, no scrolling.

python examples/dashboard_demo.py            # both ends, loopback, no hardware
python examples/follower.py --dashboard      # on the robot
python examples/leader.py   --dashboard      # on the operator's machine

For the whole architecture over a real meeting with no robot — the only setup where the network tiles and glass-to-glass populate — run the hardware-free pair. Two USB cameras on one side, nothing on the other:

export VIDEOSDK_MEETING_ID=xxxx-xxxx-xxxx VIDEOSDK_TOKEN=...
python examples/demo_follower.py             # the machine with the cameras
python examples/demo_leader.py               # anywhere

                    ── the operator's page ──
┌──────────────────────────────────────────────────────────────┐
│ teleop  leader  active                peer 0.3s         live │
├──────────────────────────────┬───────────────────────────────┤
│            wrist             │            front              │
│  wrist 30 fps                │  front 30 fps                 │   half
├──────────────────────────────┴───────────────────────────────┤
│ NETWORK                                                      │
│ latency   jitter   packet loss   bandwidth      data         │
│                                    in / out     in / out     │
│   34ms      6ms       0.4%       1840/210kbps  412/88kB/s    │   half
├──────────────────────────────────────────────────────────────┤
│ LEADER                                                       │
│ glass-to-glass                actions sent                   │
│     118ms                        50/s                        │
└──────────────────────────────────────────────────────────────┘

  the robot's page: same NETWORK band, then FOLLOWER —
  glass-to-glass · actions applied · observations · packets dropped

Off unless asked for — it binds a socket and costs CPU, which is the operator's call. --dashboard-host 0.0.0.0 reaches it on a headless robot host, and ros2 run ... -p dashboard_port:=8080 is the ROS 2 equivalent.

Every number on a page describes the machine serving it, so the band label is the provenance. Everything else the SDK measures — per-joint tracking error, obs→applied p95, correlation, tick jitter, recorder queue depth, the video-sync anchor — is still served in full at /snapshot.json.

Read DASHBOARD.md before trusting the latency figures: — means no measurement rather than zero, and glass-to-glass crosses two unsynchronised clocks.

---

Quick start, no hardware

Runnable examples live in a separate repo, videosdk-teleops-examples:

pip install -e .
git clone https://github.com/videosdk-live/videosdk-teleops-examples
cd videosdk-teleops-examples

python quickstart/protocol_demo.py --gates          # the wire format, live
python quickstart/remote_teleop.py --ticks 100      # full chain, loopback
python quickstart/my_arm.py                         # port your own arm, no hardware
python ros2/ros2_arm.py                              # same lesson, over ROS topics
python tools/inspect_session.py sessions/*  --trace 3
python lerobot-so101/export_lerobot.py sessions/*  --dry-run

remote_teleop.py runs a real follower against a real leader through an in-process transport: real gates, real clamps, real observation log. Only the network is swapped out. my_arm.py is the extension point itself: subclass Follower, implement descriptor() / read_joints() / write_joints(), and everything else — lease negotiation, the clamp, the watchdog, the e-stop latch, correlation, recording — is inherited.

On the rig

python -m videosdk_teleop.devices                 # stable by-path handles
python videosdk-teleops-examples/lerobot-so101/camera_test.py --wrist cam_a --front cam_b
python videosdk-teleops-examples/lerobot-so101/camera_test.py --servo-probe /dev/ttyACM0

export VIDEOSDK_TOKEN=... MEETING_ID=xxxx-xxxx-xxxx
cd videosdk-teleops-examples/lerobot-so101
python follower.py      # on the machine with the arm
python leader.py        # on the machine with the leader arm
python record.py        # anywhere: watches only, cannot move the arm

Nothing moves until an operator claims control, and the clamp inside tick() holds the arm the moment they stop sending. There is no wrapper to forget and no way around it — the clamp sits directly above the only call to write_joints().

follower.py / leader.py / record.py wrap the SO-101 with SO101FollowerAdapter / SO101LeaderAdapter (videosdk_teleop.adapters.lerobot), each a Follower / Leader subclass. videosdk-teleops-examples/quickstart/my_arm.py is the same three-method extension point for hardware lerobot has never heard of.

On ROS 2

One node, teleop_bridge_node, one role per host. It never owns a servo bus itself — it subscribes to your driver's joint-state topic and publishes commands to your driver's joint-command topic, and carries the gap between the two hosts over the meeting. That means the ROS topic hop between this node and your driver is part of the actuation path, not just an observability surface — full detail, including a real two-driver runbook, in ROS2.md.

Install into the interpreter ROS 2 itself runs on, not whichever one your shell points at — on Ubuntu 24.04 that needs --break-system-packages (it still only writes to ~/.local):

/usr/bin/python3 -m pip install --user --break-system-packages \
    -e '.[lerobot]'

Then build the ament package once:

ln -s "$(pwd)/ros2/videosdk_teleop_ros" ~/ros2_ws/src/
cd ~/ros2_ws && colcon build --symlink-install && source install/setup.bash
ros2 pkg executables videosdk_teleop_ros     # expect teleop_bridge_node

Now two terminals, each already running your own leader/follower driver nodes and each starting from a fresh shell that has none of the environment:

# run these in EVERY terminal
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash
export VIDEOSDK_MEETING_ID=abcd-efgh-ijkl    # both ends MUST join the same room
export VIDEOSDK_TOKEN=...                    # keep it here, not in the YAML

# 1 — follower host, relaying to your follower driver
ros2 launch videosdk_teleop_ros bridge.launch.py \
    role:=follower_side robot_id:=my_follower_arm \
    calibration_profile:=lerobot_so101_rad \
    cameras:=cam_a,cam_b labels:=wrist,front

# 2 — leader host, relaying to your leader driver
ros2 launch videosdk_teleop_ros bridge.launch.py \
    role:=leader_side robot_id:=my_leader_arm \
    calibration_profile:=lerobot_so101_rad

# 3 — look before you arm, then take control
ros2 topic echo /videosdk_bridge/remote/follower_states --once   # far arm, local units
ros2 service call /videosdk_bridge/enable std_srvs/srv/SetBool "{data: true}"

The leader logs the peer's shape as it is accepted or refused at arm time — that is the check that catches the two ends disagreeing about units, and reading one log line is cheaper than watching an arm move wrongly.

Nothing moves until that enable call. Release it with {data: false}, or stop the arm outright:

ros2 service call /videosdk_bridge/estop std_srvs/srv/Trigger        # latched
ros2 service call /videosdk_bridge/clear_estop std_srvs/srv/Trigger  # only way out, on the follower host

Watch what it is doing:

ros2 topic hz   /follower/joint_commands           # what this node is sending your driver
ros2 topic echo /videosdk_bridge/remote/follower_states
ros2 bag record /follower/joint_commands /videosdk_bridge/remote/follower_states

---

The API

Two abstract base classes, one per machine — subclass the one your machine needs. The class name says which machine; three methods (descriptor() / read_joints() / write_joints()) are the whole extension point, and everything else — lease negotiation, the clamp, the watchdog, the e-stop latch, correlation, recording — is inherited.

# ---- follower host: the machine holding the arm ----------------------------
from videosdk_teleop.adapters.lerobot import SO101FollowerAdapter

arm = SO101Follower(SO101FollowerConfig(port="/dev/ttyACM0", cameras={...}))
follower = SO101FollowerAdapter(
    "robot-lab-1", robot=arm, token=TOKEN,
    safety=SafetyConfig(slew=12.0, watchdog_timeout_s=0.5))

follower.run(hz=50)          # start(), loop, stop() -- ctrl-c is handled

# ---- leader host: the machine holding the leader arm -----------------------
from videosdk_teleop.adapters.lerobot import SO101LeaderAdapter

leader = SO101Leader(SO101LeaderConfig(port="/dev/ttyACM1"))
remote = SO101LeaderAdapter("robot-lab-1", teleop=leader, token=TOKEN,
                            max_misalignment=15.0)

leader.connect()             # local leader arm; the SDK doesn't own this bus
remote.start()
remote.claim_control()
remote.arm_when_aligned()    # hold the leader steady against the follower

for _ in remote.ticks(hz=50):
    remote.observation()     # synced joints + pixels; arms the echo too

SO101FollowerAdapter / SO101LeaderAdapter (videosdk_teleop.adapters.lerobot) are Follower / Leader subclasses that wrap an ordinary lerobot Robot / Teleoperator — see API.md §4/§5 and EXAMPLES.md §3/§4 for every method and event.

Safety is a config, not a wrapper

follower = SO101FollowerAdapter(
    "robot-lab-1", robot=arm, token=TOKEN,
    safety=SafetyConfig(
        slew=12.0,                  # max change per joint per tick, arm's own units
        watchdog_timeout_s=0.5,     # no actions for this long -> hold position
        on_starvation=FailsafeAction.HOLD))

The clamp lives directly above the only call to write_joints(), inside tick() — there is no separate wrapper object to construct and no path to the motors that skips it. It clamps every action, holds if the operator disappears, and latches on estop() until a human calls clear_estop().

on_safe_state(state, reason)	every failsafe transition
on_limited(event)	one per clamp, with magnitude
on_applied(action)	what actually reached the motors
estop() / clear_estop()	latched stop
.state / .safety	current state, limits in force

Recording

Same four methods on either side:

session = follower.start_recording("./sessions")
follower.start_episode("pick up the cube")
...                                          # your loop runs
follower.end_episode(success=True)
follower.stop_recording()

On the follower that records the arm's own observations — pixels as captured, before an encoder or a network touched them. On the leader it records the synced stream: each frame paired with joints interpolated to that frame's own capture instant. Export either with videosdk-teleops-examples/lerobot-so101/export_lerobot.py.

Correlation, for free

Leader.observation() remembers which observation it returned, and arms the next command's echo with it — so every action in a recording points at the exact observation the operator was looking at, by integer id, with no clock shared between the machines. AppliedAction.observation_id / .observation_id_local on the follower side expose the edge.

Adding a robot

Nothing here is SO-101 specific. A Follower / Leader subclass implements three abstract methods and asks no other questions of the hardware underneath — see videosdk-teleops-examples/quickstart/my_arm.py for the extension point taught end to end, and videosdk-teleops-examples/ros2/ros2_arm.py for the same lesson over ROS topics.

---

The idea

The SDK can answer both directions of one question:

For command X, which camera observation did the operator act upon? For observation Y, which action came out and what actually happened?

  FOLLOWER                                        LEADER
  mint Observation(oid=N) ──── state frame ────►  remember last_oid = N
    wrist frame 500                                       │
    front frame 900                              operator moves the arm
    robot state_seq 4410                                  │
        │                                                 ▼
  ObservationLog.get(N)  ◄──── cmd frame ──── Command(cid=M, oid=N)
    O(1), no clocks involved
        │
  AppliedAction(cid=M, oid=N, target=…, action=…, limits=[…])
        │
  RobotState(sseq=K, cid=M)

Each id is minted by the machine that owns the thing it names and only ever echoed by the other. No shared counter, no clock sync, no agreement problem.

Read SYNC.md next — it is the load-bearing document.

---

Documents

EXAMPLES.md	bring your own hardware: complete code for both ends, no SO-101 — §3/§4 are the full follower/leader tour
API.md	every public symbol: signatures, defaults, threading, exceptions — §4/§5 cover Follower/Leader
ARCHITECTURE.md	threads, modules, safety layers, adding a robot or camera
PROTOCOL.md	wire format, receive gates, what is frozen
SYNC.md	every clock, what is measurable and what is not
RECORDING.md	staging schema, drop accounting, LeRobot export
ROS2.md	teleop_bridge_node, calibration, what ROS is and is not here
DASHBOARD.md	the live stats page: every tile, its source, and what it cannot tell you
DASHBOARD_PARAMETERS.md	every dashboard number: meaning, stats key, and how it is calculated

---

Safety

Layer	Where
lease	transport, receive
sequence (duplicate / reorder)	transport, receive
staleness	control loop
watchdog	control loop, every tick
deadman	control loop
position + slew clamps	Clamp
e-stop	latched, only a human clears it

Defaults are timid on purpose. watchdog_timeout_s and max_staleness_ms should be re-derived on your link under load — the numbers that matter are the ones your network actually produces.

HOLD is the default failsafe, not TORQUE_OFF: cutting torque on a gravity-loaded arm drops it.

Every clamp is recorded with its magnitude, not just a count. A joint clamped by 0.2° is rounding; the same joint clamped by 40° means calibration is wrong, and a bare counter cannot tell them apart.

---

Testing

pip install -e ".[dev]"
pytest -q

No test touches a network, a servo, a camera, or sleep(). Everything timing-sensitive takes an explicit monotonic value. LoopbackTransport subclasses the real transport and replaces only the three methods that touch the network, so the receive gates under test are production code.

The same harness is importable for your own Follower / Leader subclasses:

from videosdk_teleop.testing import LoopbackTransport, connect, grant_lease

arm = MyArm(transport=LoopbackTransport("my-arm"))
operator = FakeLeader(transport=LoopbackTransport("operator"))  # videosdk_teleop.adapters.fake
connect(arm, operator)

---

Known limitations

1. oid is a bound, not a proof. The leader stamps the newest observation it received on the data channel; what the operator's eye saw is a video frame from a separate RTP path. Closing this needs frame metadata inside the encoded video (SEI) and vsaiortc owns the codec. Mitigated by recording both the claimed and the locally-current observation, so the doubt is measured.

2. observation → command latency is not reported. It spans two unsynchronised clocks and cannot be measured. observation → applied is exact and is the number that matters. See SYNC.md.

3. capture_mono is arrival, not shutter. True shutter timestamps are not exposed through the vsaiortc MediaPlayer path. Inter-camera skew stays meaningful; the absolute value carries a fixed unknown offset.

4. Images are re-encoded. The capture path decodes MJPG before we see it, so the recorder re-encodes JPEG on the writer thread. Off the control loop, but not free.

5. claim is first-come and unauthenticated. Any participant can take the lease. E-stop is unauthenticated by design — anyone should be able to stop the arm — but claim should be gated by meeting-level auth before this is exposed to untrusted participants.

6. SO-101 control-table addresses are unverified against your firmware. Feetech ships variants. Run camera_test.py --servo-probe once per rig.

7. videosdk does not import on Python 3.14 (pydantic v1 inside pymediasoup). The transport degrades to unavailable and everything else still runs; use 3.10–3.12 for live meetings.

8. Single-process, single-robot. No fleet, no multi-robot session, no authenticated operator handover queue.

---

Next

• Gate claim behind meeting-level identity, and add an operator handover queue
• SEI frame metadata if/when the codec is reachable — closes limitation 1
• Direct PyAV demux to keep original MJPG packets — closes limitation 4
• Bench-derived watchdog_timeout_s / max_staleness_ms from a probe script
• Bus-budget measurement to justify raising observation_hz toward camera fps
• Policy playback: replay a LeRobot dataset through the same safety envelope