rerun-lerobot 0.2.0

Convert Rerun RRD recordings into LeRobot v3 datasets.

         

rerun-lerobot

rerun-lerobot is an official Rerun package for converting Rerun RRD recordings into LeRobot v3 datasets.

rerun-lerobot uses the Rerun catalog API to query and transform recordings into the LeRobot v3 format used for imitation-learning training pipelines in PyTorch. The source can be a local directory of RRD files (served by the OSS Rerun server) or a remote Rerun catalog. It infers data types from the recordings, resamples all time series to a target frame rate, and writes a LeRobot v3 dataset. Video streams are efficiently remuxed without re-encoding.

Alternatives

You can also train directly from the free Rerun OSS Server and on the commercial Rerun Hub. This will generally be a lot simpler and faster than first converting to LeRobot. See these docs for how.

Installation

pip install rerun-lerobot

⚠️ Dependency conflict with LeRobot

The conversion relies on the Rerun OSS server API (rr.server.Server), which requires rerun-sdk >= 0.27. LeRobot currently pins rerun-sdk < 0.27, so a naive install will fail to resolve. Override LeRobot's pin at install time (and pull the matching datafusion via the [datafusion] extra — a plain rerun-sdk upgrade resolves datafusion too new for the catalog client).

With uv, in a project's pyproject.toml:

[tool.uv]
override-dependencies = ["rerun-sdk[datafusion]>=0.27"]

Or on the command line (note: uv run --with does not apply overrides — use uv pip install):

uv pip install rerun-lerobot --override <(echo "rerun-sdk[datafusion]>=0.27")

With pip, install and then force the newer rerun-sdk (the resolver will warn about LeRobot's pin; that is expected):

pip install rerun-lerobot
pip install --upgrade "rerun-sdk[datafusion]>=0.27"

Usage

The package installs a rerun-lerobot CLI that converts recordings into a LeRobot v3 dataset. Exactly one source is required: a local directory of RRD files (--rrd-dir), a remote Rerun catalog server plus dataset name (--catalog-url), or a full Rerun dataset URL (--dataset-url).

From a directory of RRD recordings:

rerun-lerobot \
  --rrd-dir /path/to/recordings \
  --output /path/to/output/dataset \
  --dataset-name my_robot_dataset \
  --fps 10 \
  --index real_time \
  --action /action:Scalars:scalars \
  --state /observation/joint_positions:Scalars:scalars \
  --task /language_instruction:TextDocument:text \
  --video front:/camera/front

From a Rerun catalog server (looked up by --dataset-name, optional --catalog-token for auth):

rerun-lerobot \
  --catalog-url rerun+http://my-catalog-host:51234 \
  --dataset-name my_robot_dataset \
  --catalog-token "$RERUN_TOKEN" \
  --output /path/to/output/dataset \
  --fps 10 \
  --index real_time \
  --action /action:Scalars:scalars \
  --state /observation/joint_positions:Scalars:scalars \
  --video front:/camera/front

Directly from a Rerun dataset URL (bundles the catalog server and dataset id — no --dataset-name needed; --catalog-token still applies for auth):

rerun-lerobot \
  --dataset-url rerun://hostname:443/entry/18B40C6FA7631F942c0e90030ac230fa \
  --output /path/to/output/dataset \
  --fps 10 \
  --index real_time \
  --action /action:Scalars:scalars \
  --state /observation/joint_positions:Scalars:scalars \
  --video front:/camera/front

Guided start: discovering columns

You don't need to know the exact column names up front. Start with just a source and an output:

rerun-lerobot \
  --dataset-url rerun://hostname:443/entry/18B40C6FA7631F942c0e90030ac230fa \
  --output /tmp/lerobot

Because --action, --state, and --fps are missing, the tool connects to the dataset, prints the convertible columns it found — action/state candidates (numeric vectors, with dimensions), timelines for --index, task-text candidates, and video streams — and suggests a full command to copy, edit, and re-run. Pass --inspect to do this explicitly without converting.

Action / state candidates (numeric vector columns):
  /robot/action:Scalars:scalars          dim 7    [Scalars]
  /observation/joints:Scalars:scalars     dim 6    [Scalars]
  ...
Timelines (for --index):
  log_time      (timestamp[ns])
  ...
Suggested command:
  rerun-lerobot --dataset-url ... \
    --output /tmp/lerobot \
    --fps 10 \
    --index log_time \
    --action /robot/action:Scalars:scalars \
    --state /observation/joints:Scalars:scalars

The action/state picks are best-guesses (by name, else the first candidates) — review them: the tool cannot know which numeric column is the commanded action vs the observed state.

Column specification format

Action, state, and task columns are specified as fully qualified columns:

entity_path:ComponentName:field_name

For example /robot/action:Scalars:scalars.

Video specification format

Videos are specified as key:path:

• key: camera identifier (e.g. front, wrist)
• path: entity path to the video stream (e.g. /camera/front)

The converter expects a VideoStream archetype at the specified paths.

Camera sources and --output-format

Cameras (--video key:path) can be stored in the recording as any of:

• VideoStream — compressed video packets (H.264 / HEVC / AV1)
• EncodedImage — per-frame JPEG or PNG
• Image — raw pixel buffers

The archetype is detected automatically. Unsupported archetypes/codecs abort the conversion with a message telling you what was found.

LeRobot can only store two things: PNG image frames (dtype: "image") or an MP4 video (dtype: "video", codec H.264 / HEVC / AV1). Choose with --output-format:

--output-format	Result
(omitted)	Keep the source format if LeRobot can store it, else H.264 (see below)
png	PNG image frames
h264 / hevc / av1	MP4 video in that codec

jpg is rejected — LeRobot has no per-frame JPEG storage; use png or a video codec.

Default (keep-original), per camera:

Source	Default output
video H.264 / HEVC / AV1	same codec, remuxed (copied, no re-encode)
EncodedImage PNG	png
EncodedImage JPEG	h264 (re-encoded — jpeg isn't storable)
raw Image	h264

Remux vs re-encode. When a video camera's source codec already equals the output codec, packets are copied straight into MP4 — fast and lossless — provided the source frame rate is within 5% of --fps. Otherwise (codec change, image sources, or fps mismatch) frames are decoded and re-encoded (video) or written as PNG (image).

Resampling. --fps resamples the scalar streams (action / state / task) — output rows are the --index timeline frames where the action column is present. Each camera frame is sampled at the nearest source frame at-or-before that row's timestamp (latest-at). The frame shape (height, width, 3) is inferred by decoding one frame; all frames are converted to RGB.

Full example

rerun-lerobot \
  --rrd-dir ./robot_recordings \
  --output ./lerobot_dataset \
  --dataset-name robot_demos \
  --fps 15 \
  --action /robot/action:Scalars:scalars \
  --state /robot/state:Scalars:scalars \
  --task /task:TextDocument:text \
  --video front:/camera/front \
  --video wrist:/camera/wrist \
  --action-names "joint_0,joint_1,joint_2,gripper" \
  --state-names "joint_0,joint_1,joint_2,gripper"

The output directory contains:

• data/: Parquet files with aligned time series data
• videos/: encoded video files (for video-output cameras)
• images/: PNG frames (for png-output cameras)
• meta/: dataset metadata and episode information

Python API

from pathlib import Path

from rerun_lerobot import LeRobotConversionConfig, VideoSpec
from rerun_lerobot.lerobot.export import (
    convert_catalog_dataset_to_lerobot,
    convert_dataset_url_to_lerobot,
    convert_rrd_dataset_to_lerobot,
)

config = LeRobotConversionConfig(
    fps=15,
    index_column="real_time",
    action="/robot/action:Scalars:scalars",
    state="/robot/state:Scalars:scalars",
    task="/task:TextDocument:text",
    videos=[VideoSpec(key="front", path="/camera/front")],
)

# From a local directory of RRD files:
convert_rrd_dataset_to_lerobot(
    rrd_dir=Path("./robot_recordings"),
    output_dir=Path("./lerobot_dataset"),
    dataset_name="robot_demos",
    repo_id="robot_demos",
    config=config,
)

# ...or from a remote Rerun catalog:
convert_catalog_dataset_to_lerobot(
    catalog_url="rerun+http://my-catalog-host:51234",
    dataset_name="robot_demos",
    token=None,  # or an auth token
    output_dir=Path("./lerobot_dataset"),
    repo_id="robot_demos",
    config=config,
)

# ...or straight from a Rerun dataset URL:
convert_dataset_url_to_lerobot(
    dataset_url="rerun://hostname:443/entry/18B40C6FA7631F942c0e90030ac230fa",
    token=None,  # or an auth token
    output_dir=Path("./lerobot_dataset"),
    repo_id="robot_demos",
    config=config,
)

Both delegate to convert_dataset_to_lerobot(dataset, ...), which works on any connected rerun.catalog.DatasetEntry if you already have one.

To discover columns before building a config (the same guidance the CLI prints), use the matching inspect_* function — each returns a DatasetInspection you can read or format:

from rerun_lerobot.lerobot.export import inspect_dataset_url

inspection = inspect_dataset_url(
    "rerun://hostname:443/entry/18B40C6FA7631F942c0e90030ac230fa",
    token=None,
)
print(inspection.format_report())

for candidate in inspection.action_state_candidates:
    print(candidate.name, candidate.dim, candidate.component)

action_guess, state_guess = inspection.guess_action_and_state()
index_guess = inspection.guess_index()

There is also inspect_catalog_dataset(...), inspect_rrd_dataset(...), and inspect_dataset(dataset) for an already-connected DatasetEntry.

Running locally (without publishing to PyPI)

To run the rerun-lerobot CLI straight from a checkout of this repo:

uv sync --dev          # create .venv with the package installed (editable)
uv run rerun-lerobot --help

uv run executes the entry point from the local source — no build or PyPI upload needed, and edits to the code take effect immediately. Alternatively, activate the environment and call the binary directly:

source .venv/bin/activate
rerun-lerobot --help

Or, without cloning, install into a throwaway virtualenv with the override applied (uv run --with can't override LeRobot's transitive rerun-sdk pin, so use uv pip install --override):

uv venv /tmp/rl-venv
uv pip install --python /tmp/rl-venv \
  "git+https://github.com/rerun-io/rerun-lerobot" \
  --override <(echo "rerun-sdk[datafusion]>=0.27")
/tmp/rl-venv/bin/rerun-lerobot --help

Development

uv sync --dev
uv run ruff format --check
uv run ruff check
uv run mypy
uv run pytest

The end-to-end test (tests/test_e2e.py) downloads a small public RRD sample and runs a full conversion; it is cached under tests/data/ and skips automatically when offline.

License

Licensed under either of Apache-2.0 or MIT at your option.