ROS Communication DevContainer

The ROS Communication DevContainer is a Docker-based solution designed to streamline the bidirectional synchronization of ROS2 topics between two Linux machines. It provides built-in compression and routing capabilities for over-the-air (OTA) data transfer: selected topics are remapped into an OTA namespace and transmitted either via direct DDS (CycloneDDS) or through a Zenoh router. When desired, the session can also place local application nodes and OTA-facing bridge nodes into separate ROS 2 domain IDs and automatically generate a standard ROS 2 domain_bridge configuration for the /com/... boundary. This project aligns with the publication “Scalable Remote Operation for Autonomous Vehicles: Integration of Cooperative Perception and Open Source Communication.”

Getting Started

Prerequisites

• Docker installed on all machines
• Git for configuration management
• Python 3 with virtual environment support (python3-venv on Debian/Ubuntu; for versioned Python packages this may be named like python3.14-venv)
• ros2docker v0.1.4 or newer. The local installer below installs the pinned supported range into this checkout's virtual environment.
• tmux for the optional rosotacom scenario orchestration commands
• Machines connected to the same network (VPN or local WLAN)

Convenience CLI: rosotacom

This repository's main entrypoint for starting ROS communication sessions is the checkout-local rosotacom command. Each checkout owns its own .venv, so multiple rosotacom versions can coexist without global symlink drift.

Install

Use a released version, or develop from a checkout — both with off-the-shelf tooling, nothing rosotacom-specific:

# released version (recommended for users)
pipx install rosotacom-dev        # isolated, on PATH everywhere

<!-- The distribution is rosotacom-dev; the import name and the CLI stay
     `rosotacom`. This fork publishes under its own name so that `rosotacom`
     on PyPI stays reserved for the FZI upstream. -->
#   or: pip install --user rosotacom

# a specific development build (for consumers tracking work in progress)
pipx install "rosotacom-dev==2.4.dev3"

# from a source checkout (developers)
cd /path/to/ros_communication_devcontainer
./install.sh                      # builds a checkout-local .venv
source .venv/bin/activate         # use it in this terminal
#   ./install.sh --global-symlink # optional: put it on PATH via ~/.local/bin

rosotacom --version
python -m rosotacom --version
rosotacom doctor

Every commit that lands on the development branch with green CI is published as a X.Y.devN pre-release, so a consumer never has to wait for a release decision or reach past the package to a checkout. pip ignores pre-releases unless you pin one exactly, so this cannot reach you by accident. See docs/release.md.

For a source checkout, that single source .venv/bin/activate command also enables completion. Nothing needs to be added to your shell configuration.

For a pipx/global installation, add one line once to your shell startup file:

# zsh: add this line to ~/.zshrc
eval "$(rosotacom completion zsh)"

# bash: add this line to ~/.bashrc
eval "$(rosotacom completion bash)"

Every new terminal then loads completion automatically; you do not run the command manually in each terminal. After reloading the shell, rosotacom <TAB><TAB> lists commands, rosotacom smoke <TAB><TAB> lists sessions from the active project, and a prefix such as rosotacom smoke 1<TAB> expands to 1_heartbeat. The same session completion is available for start, stop, status, test, and the probe commands; absolute and relative session-directory paths still complete normally. --identity <TAB><TAB> lists the peer identities from the selected session or scenario. --peer <TAB><TAB> completes logical peer keys, and --peer a=<TAB><TAB> completes named hosts from the active deployment file. --peer-address <TAB><TAB> completes logical peer keys for raw overrides. Scenario attach and stop complete only active scenarios and identities. Running rosotacom completion without a shell argument infers bash or zsh from $SHELL.

Completion is not pinned to the version that registered it: each Tab press runs the rosotacom currently selected by PATH. Activating another checkout's virtual environment therefore switches both the command and its completion, and deactivate switches back without a reset step. A pipx-suffixed executable can be registered independently with, for example, eval "$(rosotacom@2.2.0 completion zsh)".

Multiple versions / try-without-disturbing. Because each checkout owns its own .venv and ./install.sh never touches your PATH on its own, you can keep a pinned/production version in use while smoke-testing another in one terminal:

cd ~/checkouts/rosotacom-newest   # a second checkout (or git worktree)
./install.sh                      # builds .venv here; your other rosotacom is untouched
source .venv/bin/activate         # this terminal now runs the new commit
rosotacom smoke                   # built-in example, zero config
deactivate                        # back to your usual rosotacom

For coexisting released versions, pipx install rosotacom-dev==2.2.0 --suffix=@2.2.0 gives you a rosotacom@2.2.0 alongside the default — standard pipx, no bespoke version manager.

Basic Setup

rosotacom uses five layers of configuration/runtime state:

• A project setup file (rosotacom.yaml) points to host-local resources such as ros2docker.json, static sessions/, ignored session-instances/, and an optional deployment file.
• A session config defines the communication behavior for one run: logical peers, topics, QoS, processing, and transport choices.
• An optional scenario config composes one communication session with identity-specific local application containers.
• An optional deployment config maps named physical hosts to reachable addresses and optional SSH targets.
• A session instance stores one concrete run: generated config, catmux pane logs, smoke debug output, and future rosbags.

rosotacom resolves the active rosotacom.yaml by scope (first wins), using the shell / local / global vocabulary from pyenv:

1. --project <path> (alias of --rosotacom-config) on the command
2. shell — ROSOTACOM_CONFIG in the environment (needs eval, since only your shell can change its own environment)
3. local — the nearest rosotacom.yaml discovered upward from the cwd
4. global — a machine-wide default (rosotacom config set project ... --global)
5. a built-in example project, so a fresh install runs with zero setup

The built-in example is used in place (read-only) and writes nothing to $HOME: its generated session output goes to a per-user tmpfs dir under $XDG_RUNTIME_DIR (printed on start/smoke), so it is ephemeral. For persistent, editable projects, create one with rosotacom examples create.

So the simplest workflow is just to be in a project directory (the local scope — no command needed, just have the file):

rosotacom examples create ./rosotacom_examples
cd ./rosotacom_examples
rosotacom doctor                # picks up ./rosotacom.yaml automatically

Inspect or pin the selection with rosotacom config:

rosotacom config show                                            # every scope + the winner
rosotacom config set project ./rosotacom.yaml --global           # machine-wide default
eval "$(rosotacom config set project ./rosotacom.yaml --shell)"  # this terminal only

The copied packaged example project uses this layout:

rosotacom.yaml
ros2docker.json
deployment.example.yaml
sessions/
scenarios/
session-instances/
scripts/

session_configs_dir and scenario_configs_dir are ordered YAML lists. Put project-local directories first and shared/example directories later when one project should discover both.

Locating packaged resources

rosotacom is installed in an isolated environment (pipx, or a project venv), so another tool cannot import rosotacom from a system Python to find the files that ship with it. Ask the CLI instead — it prints one absolute path, so it composes directly in a shell:

rosotacom resources path com_msgs   # the ROS 2 message package
rosotacom resources path ws         # the packaged runtime workspace

Use this wherever another project needs rosotacom's files on the host, for example to bake com_msgs into its own container image. Ask for the resource by name rather than reconstructing a path: the names are part of the CLI contract, the layout behind them is not.

See the example project README for the copyable example layout, the session configuration reference for the session schema, the deployment configuration reference for the machine-specific schema and peer-binding precedence, terminology.md for the project vocabulary, and CONCURRENCY.md for what can run in parallel and how conflicting runs are detected and aborted.

The normal examples intentionally contain no machine addresses. rosotacom smoke injects isolated Docker-network addresses, while manual and OTA runs select physical hosts explicitly.

Write or edit session configs under sessions/<name>/:

• session-definition.yaml for a self-contained session
• session-parametrization.yaml for a template plus parameters

Run rosotacom on each peer with the same active setup but a different identity:

# on peer "a"
rosotacom start 1_heartbeat --identity a \
  --peer-address a=10.0.0.10 --peer-address b=10.0.0.11

# on peer "b"
rosotacom start 1_heartbeat --identity b \
  --peer-address a=10.0.0.10 --peer-address b=10.0.0.11

rosotacom reads the static session input and creates generated files under session-instances/<date>/<session>_<timestamp>_<id>/config/, including per-peer plugin/session specs, topic lists, optional QoS, and optional domain_bridge.yaml. Catmux pane output is logged under the same instance in logs/<peer>/catmux/.

Complete use cases with scenarios

A session intentionally describes only the communication contract. When a use case also needs a local ROS application, rosbag, or ROS command, define a scenario above it:

schema_version: 1
session: 2_native_chatter

applications:
  a:
    - name: native_application
      ros2docker_config: ../../scripts/2_native_chatter/machine_a/external.ros2docker.json
  b:
    - name: native_application
      ros2docker_config: ../../scripts/2_native_chatter/machine_b/external.ros2docker.json

Start one identity's communication and local application together:

rosotacom scenario start 2_native_chatter --identity a
rosotacom scenario list
rosotacom scenario attach
rosotacom scenario stop

scenario list shows both configured scenarios and currently active scenario/identity pairs. attach and stop infer omitted values when exactly one active choice exists; otherwise their completions and error messages show the eligible active choices.

The outer view uses an isolated host tmux server with one full window for communication and one full window per local application. Its prefix remains Ctrl-b; switch windows with Ctrl-b n/Ctrl-b p, and send the inner container-side catmux prefix with Ctrl-b Ctrl-b. Detaching from the outer tmux does not stop the use case. scenario stop owns cleanup of the application containers, communication container, and outer tmux session.

Safe Replay Testing with Anonymized Rosbags

To test scenarios involving proprietary data safely, first record the processed outbound handoff topics that the session pipeline actually forwards into the OTA path. rosotacom anonymize consumes that processed trace bag and the matching session/scenario definition. It:

1. Resolves the session pipeline and selects the outbound handoff topics that feed /com/out/... and /ota/....
2. Fails closed if the input bag only contains raw source topics and is missing the processed handoff topics.
3. Renames the handoff topics to generic names like /topic1, /topic2, etc.
4. Replaces message contents with mock payloads while preserving message shape, timestamps, payload lengths, bag QoS metadata, and playback QoS overrides. For ffmpeg camera packets it additionally preserves the keyframe flag and pts, keeping the GOP burst structure analyzable — see docs/ffmpeg-keyframes.md.
5. Generates a self-contained rosotacom project containing the anonymized bag, processed-topic session definition, replay scenario, QoS overrides, and an anonymization manifest.

rosotacom anonymize /path/to/processed_handoff_bag \
  -s 3_comp_occ_grid \
  -o ./anonymized_project

Usage Examples

Create the example project first (cwd discovery wires it automatically once you cd in):

rosotacom examples create ./rosotacom_examples
cd ./rosotacom_examples

Run the local heartbeat smoke test:

rosotacom smoke

To smoke-test one specific configured session locally, pass its name:

rosotacom smoke 1_heartbeat

The smoke test verifies both directions through the communication path: it waits for /com/in/a/heartbeat_a and /heartbeat_a in peer b, plus /com/in/b/heartbeat_b and /heartbeat_b in peer a. For each checked topic it also reports a SMOKE_METRIC line with the received rate (hz) and end-to-end latency (delay_s) so rate and latency regressions are visible. It prints the session-instances/... artifact path so failures (and the per-peer logs/<peer>/catmux/... pane output) can be inspected after the containers stop.

For local debugging, run the same end-to-end idea interactively:

rosotacom smoke 2_native_chatter --interactive
rosotacom smoke --interactive --list
rosotacom smoke 2_native_chatter --interactive --stop

Interactive smoke starts all peers on one host, opens an isolated outer tmux session, and keeps full windows for each peer's communication container. If the target is a scenario, it also opens one window per local application container; each application shares its peer communication container's isolated network namespace so local ROS discovery behaves like colocated processes. A verification window runs the local checks/status view while you inspect the system. Its upper pane keeps the verification log; its lower pane watches live status. The outer prefix is Ctrl-b; use Ctrl-b n/Ctrl-b p for windows and Ctrl-b Ctrl-b for the inner catmux session.

rosotacom smoke TARGET --interactive accepts either a session or a scenario. When both names exist, interactive smoke treats TARGET as a scenario unless you pass --target-type session. Non-interactive rosotacom smoke TARGET keeps its existing session-only behavior.

Local smoke runs of different targets are parallel-safe: every run gets instance-scoped container names and its own isolated Docker network. Starting a run that cannot coexist with an active one (same smoke target, same session identity, any benchmark) aborts immediately with the conflicting containers and the stop command. rosotacom ps shows what is active in this workspace; CONCURRENCY.md documents the boundaries.

For a real two-host OTA check with no deployment file, provide both addresses and only the SSH target needed to reach the remote peer:

rosotacom ota-smoke 2_native_chatter \
  --peer-address a=10.0.0.10 \
  --peer-address b=10.0.0.11 \
  --peer-ssh b=robot-b

For reusable host names, reference one deployment file from rosotacom.yaml:

# rosotacom.yaml
deployment: deployment.yaml

# deployment.yaml
hosts:
  workstation:
    address: 10.0.0.10
    ssh: null
  robot:
    address: 10.0.0.11
    ssh: robot-b

rosotacom ota-smoke 2_native_chatter \
  --peer a=workstation \
  --peer b=robot

ota-smoke accepts sessions and scenarios. It puts rosotacom on each peer, stages the active project, runs delivery and isolation checks, collects artifacts, stops the run, and removes the remote workdir. For scenario targets the checks start only once every declared application container is running (bounded at 10 minutes, covering a cold peer's first image build); an application that never comes up fails the run by name instead of as missing publishers. Add --interactive for a local control tmux with one attachable communication/catmux window per peer with a live status pane below each one. Scenario targets also get one native application-container window per scenario app. Stop an interactive run with rosotacom ota-smoke TARGET --interactive --stop; pass the same peer/deployment arguments if the local tmux metadata is not available. Use --keep-running to leave components up, --keep-workdir to retain staged files, or --reuse to reuse an existing installation.

What the peers run: --install-mode

# default: stage and install what you are running, uncommitted changes included
rosotacom ota-smoke 2_native_chatter --peer a=workstation --peer b=robot

# rehearse a deployment: the peers install the published artefact
rosotacom ota-smoke 2_native_chatter --peer a=workstation --peer b=robot \
  --install-mode pin                # defaults to your own version
rosotacom ota-smoke 2_native_chatter --peer a=workstation --peer b=robot \
  --install-mode pin --install-pin 2.4

# test the machines as they are: no staging, no install, their own checkouts
rosotacom ota-smoke 2_native_chatter --peer a=workstation --peer b=robot \
  --peer-ssh b=robot-b --install-mode checkout \
  --peer-checkout a=/home/me/fleet_mgmt --peer-checkout b=/home/robot/fleet_mgmt

source is right while iterating on rosotacom itself — the peers run the code in front of you. pin is right when the run is meant to stand in for a real deployment: the peers install from the index, so a packaging defect (a file missing from the wheel, an undeclared dependency) fails in the rehearsal instead of on the target machine. source cannot catch that class of defect by construction, because it never builds the artefact that ships.

checkout answers a different question again: not "does this artefact work" but "is this machine ready". Nothing is staged and nothing is installed — each peer runs the rosotacom it already has, from the project checkout it already has, so what is under test is the state git and the machine's own install left behind, dependency pins included. Ground truth moves into the repository: push, pull, run. It is also the only mode in which a project may reference anything outside its own directory — staging copies the project directory, so a mount or config path leading above it cannot survive the copy and fails on the peer as a missing path.

Because the same repository sits at a different absolute path on every machine, --peer-checkout is required for every peer; the path inside the repository is taken from your own checkout, so the project has to live in one. Preparation becomes verification: each peer is asked whether the checkout, the project file and rosotacom are there, and each version is printed. --cleanup does nothing in this mode — there is no staging to undo, and the workdir is your repository.

The run manifest records which mode and which version the peers ran, so a result can be attributed to an artefact rather than to a working copy.

How the peers are reached: --peer-exec

--peer-ssh gives the orchestrator a shell on the peer, which is more than a run needs and more than some machines should hand out. --peer-exec names the transport instead:

rosotacom ota-smoke 2_native_chatter --peer a=workstation --peer b=robot \
  --install-mode checkout \
  --peer-checkout a=/home/me/fleet_mgmt --peer-checkout b=/home/robot/fleet_mgmt \
  --peer-exec b='remote-rosotacom robot'

The command is run with the script as its final argument — the contract ssh host <script> already has — so anything of that shape can carry a peer: a wrapper that logs what a run asks for, a container exec, or a forced command that accepts some scripts and refuses the rest. rosotacom does not know or care which; it only stops assuming that reaching a peer means having a shell on it.

Two restrictions, both load-bearing. It applies to --install-mode checkout only, because the staging modes push a tar stream and a file write by taking the ssh client's own argv apart, and a transport that promises no more than "run this script" cannot carry them. And a peer takes either --peer-ssh or --peer-exec, never both: leaving it ambiguous hides which one carried the run.

-t and -o BatchMode=yes are dropped rather than forwarded. They are options of one client, and a transport that is not ssh never agreed to understand them.

Live status / debugging overview

Enable a continuously-updated, per-topic pipeline overview by setting shared.use_status_overview: true in the session definition (see the ready-made 1_heartbeat_status example). For every configured topic it tracks where the topic currently is in the communication pipeline (furthest stage reached and the first stage that is missing/broken), plus last-message age, Hz, mean size, latency, exact wrapped-topic sequence loss/reordering, and echo-derived RTT/clock offset.

The running session writes, under session-instances/.../logs/<peer>/status/:

• status.json — machine-readable snapshot (source of truth) for tools/agents, refreshed on a short interval and on every state transition,
• status.txt — a human-rendered table, and
• events.jsonl — state transitions plus per-(topic, seq) transit records for wrapped topics (delivered/lost/reordered, section latency, size, inter-arrival, and jitter).

For fixed-interval network-condition samples alongside the same session, enable the link trace recorder. It writes link_trace.jsonl under the same status directory with /proc/net/dev counter deltas, echo-heartbeat RTT/loss provenance, and an optional modem-metrics hook.

Convert a recorded link trace into a profiles-file entry when you want to replay the same drive conditions in a repeatable benchmark:

rosotacom profile from-trace session-instances/.../logs/a/status/link_trace.jsonl \
  --mode timeline --name drive_replay --out generated-profiles.yaml
rosotacom benchmark recovery --profiles-file generated-profiles.yaml --profile drive_replay

Timeline mode emits piecewise-constant RFC 0004 segments, turns long sample gaps into outage: reconnect, and turns sustained 100% probe loss into outage: catchup. Static mode distills one profile with median valid rate, p90 one-way delay, delay-spread jitter, and mean loss; constrain it with --window START:END when only part of a drive should calibrate the profile. RTT is mapped to symmetric one-way delay. Passive /proc/net/dev throughput is only a lower-bound observation, so rate is emitted only when a sample is marked as saturated/probed/capacity-like; otherwise the converter omits rate instead of fabricating capacity. The generated YAML carries a provenance comment with the source path, SHA-256, parameters, and these caveats.

Render an offline route map when a run also has GPS or pose samples:

rosotacom geomap --bag session-instances/.../rosbags/a/native --gps-topic /fix \
  --trace session-instances/.../logs/a/status/link_trace.jsonl \
  --metric observed_tx_kbps \
  --out-csv artifacts/geo-link-quality.csv \
  --out-html artifacts/geo-link-quality.html

geomap writes georeferenced CSV samples, an HTML report, and a sibling .route.png route image. The timestamp offset between trace/event time and bag/GPS time is explicit; see docs/geo-link-map.md for the alignment model, CSV fixture workflow, and supported metrics.

Read it from the host with the status command:

rosotacom status 1_heartbeat_status            # human-readable table
rosotacom status 1_heartbeat_status --json     # machine-readable, for tools/agents
rosotacom status 1_heartbeat_status --watch    # live refresh

Join one or more peer timelines after a run:

rosotacom metrics session-instances/.../logs/a/status/events.jsonl \
  session-instances/.../logs/b/status/events.jsonl

Use --records for joined per-message rows instead of the loss and p50/p95 summary. Wrapped-topic corrected latency requires shared.use_heartbeat: true; status.json.clock_sync reports the selected minimum-RTT sample, peer offset, sample age, and symmetric-path assumption. Uncorrected OTA delay remains a separate field.

Compare how a stream changes across recorded stages — for example native pre-processing, processed handoff, and post-OTA transit rows:

rosotacom stream-stats \
  --bag pre=logs/b/metrics/stages_20260702_172900:/sensors/camera/front_medium/resized/image_rect_color \
  --bag handoff=logs/b/metrics/stages_20260702_172900:/sensors/camera/front_medium/resized/image_rect_color/compressed/restamped/drop1of2/ffmpeg \
  --events post=logs/b/status/events.jsonl:/sensors/camera/front_medium/resized/image_rect_color \
  --out stream-stats

The command emits a side-by-side Markdown table plus stream-stats.json, with per-stage size distributions, observed rate, interval regularity, and FFMPEG GOP/keyframe shape when available. MCAP bag sources use the Python mcap reader; other rosbag2 storage backends require rosbag2_py. events sources work from the recorded RFC 0003 transit rows. See docs/ffmpeg-keyframes.md for the GOP methodology and example camera interpretation.

Turn a whole recorded instance into an explanation of where and how delivery degraded — loss bursts, latency excursions, and rate collapses with exact boundaries, joined with link-trace/profile/keyframe context:

rosotacom report session-instances/<day>/<instance>

It writes report/report.json, report/report.md, and per-stream figures with the events marked (needs the [plots] extra). See docs/forensics-report.md for the detection semantics and the correlation-not-causation caveat.

For opt-in local per-step latency, record generated local stage topics:

shared:
  use_status_overview: true
  metric_backbone:
    record_stages: true

The MCAP is written under logs/<peer>/metrics/. Analyze an ordered pipeline with ros2 run com_py stage_latency BAG TOPIC....

For decoded camera stages, compute offline image quality with PSNR/SSIM:

rosotacom videoquality \
  session-instances/.../logs/b/metrics/stages_... \
  session-instances/.../logs/a/metrics/stages_... \
  --ref-topic /camera/image \
  --degraded-topic /camera/image/ffmpeg/raw \
  --out videoquality.json \
  --plot videoquality.png \
  --min-mean-psnr 30 \
  --min-mean-ssim 0.90 \
  --max-loss-pct 0

Use the outbound pre-transport image topic as the reference (native when no preprocessing runs; otherwise the generated topic immediately before transport, such as a restamped/drop output) and the receiver-side decoded native_in stage as the degraded stream. The reference topic may be decoded sensor_msgs/msg/Image or JPEG/PNG sensor_msgs/msg/CompressedImage; the receiver-side ffmpeg comparison topic should be the decoded /raw reverse transport output. Lost frames are reported as delivery loss and are never averaged into PSNR/SSIM. Encoded ffmpeg_image_transport_msgs/msg/FFMPEGPacket topics carry packet bytes, not pixels, so select the decoded /raw topic when computing image quality. The same command also accepts JSON frame manifests for host-only checks:

rosotacom videoquality --make-synthetic /tmp/rosotacom-quality
rosotacom videoquality \
  /tmp/rosotacom-quality/reference-frames.json \
  /tmp/rosotacom-quality/degraded-frames.json \
  --min-mean-psnr 20 --max-loss-pct 0

After a recording run, validate that a session instance still contains the artifacts needed for later analysis:

rosotacom bundle check session-instances/.../drive_run --peer a --peer b \
  --file config/resolved-session.yaml \
  --bag rosbags/a/native

bundle check validates each expected peer's status.json and events.jsonl, requires transit rows in the event streams, and validates expected rosbag2 bags from their metadata.yaml message counts. Extra expected artifacts can be passed as repeated --file, --optional-file, --bag, and --optional-bag flags, or via a generic YAML manifest:

schema_version: 1
peers: [a, b]
required_files:
  - config/resolved-session.yaml
required_bags:
  - path: rosbags/a/native
    label: native bag
optional_files:
  - traces/link.jsonl

Phase 1 samples local-domain stages directly. OTA stages are graph-only and their activity is inferred from adjacent local flow, so the status overview never creates an additional OTA payload subscription. Combine both peers' files for the full end-to-end picture; cross-peer confirmation is reserved for a later phase.

Run the CI single-machine smoke matrix locally:

just test-e2e-smoke

Run the heartbeat example manually:

./scripts/1_heartbeat/run_machine_a.sh
./scripts/1_heartbeat/run_machine_b.sh

Example 2 is the pilot for one-command scenario orchestration:

rosotacom scenario start 2_native_chatter --identity a

The existing run_external.py and run_communication.sh helpers remain available as a compatibility fallback. Examples 3–6 continue to use those helpers for now.

The sessions/ directory contains curated built-in session definitions:

• 1_heartbeat: minimal heartbeat exchange
• 1_heartbeat_status: heartbeat exchange with the live status overview enabled
• 2_native_chatter: bridge /chatter from machine_b to machine_a
• 3_comp_occ_grid: compressed occupancy grid over DDS
• 4_comp_occ_grid_zen: compressed occupancy grid through Zenoh
• 5_sized_payload: sized payload test over DDS
• 6_sized_payload_zen: sized payload test through Zenoh
• 14_remote_assist_anonymized: generic processed replay shape for the remote-assist OTA contract; run it with rosotacom smoke 14_remote_assist_anonymized
• 15_remote_assist_anonymized_costmap: single-stream cut of example 14 — only /topic5, the anonymized compressed occupancy-grid stream; run it with rosotacom smoke 15_remote_assist_anonymized_costmap
• 16_remote_assist_anonymized_camera: single-stream cut of example 14 — only /topic9, the anonymized ffmpeg camera stream; run it with rosotacom smoke 16_remote_assist_anonymized_camera
• 17_synthetic_camera_quality: synthetic raw camera stream encoded through ffmpeg and reverse-republished as decoded /raw frames for offline rosotacom videoquality checks

Transport-combination coverage lives under tests/sessions/rmw_matrix and is generated from tests/sessions/generate_rmw_matrix.py.

rosotacom benchmark uses the packaged bench_1_* sessions for local or OTA benchmark probes. Benchmark sessions default to Cyclone DDS; pass --rmw fastdds, --rmw cyclone, or another supported session RMW value to pin a run. Each run writes a self-contained result.json under its benchmark artifact directory with the selected RMW, configured load and offered bandwidth, profile shaping context, thresholds, verdict, per-topic loss/latency/jitter metrics, and the runner fingerprint. Deterministic runs gate against the committed two-sided performance bands in budgets.jsonl via rosotacom benchmark compare, and bands move only through rosotacom benchmark ratchet — see docs/performance-bands.md for the band schema, verdicts, and the ratchet workflow (RFC 0007).

Two result sets taken in different environments — emulated versus a real link, or shaped versus unshaped — are compared with rosotacom benchmark delta:

rosotacom benchmark delta emulated/ ota/ \
  --label-reference emulated --label-measured ota \
  --out delta.json --markdown delta.md

It pairs runs on genre, profile, RMW and session name, so a row that ran in one environment and not the other is reported as unmatched rather than silently dropped — "it got faster" and "it did not run" must not look the same. Every report is labelled monitor_only and carries each side's command and profile file, so a table pasted into a paper or an issue can be traced back without the directory it came from.

It exits 0 whatever the numbers say, and non-zero only when the two sets share no rows at all — a delta is an observation to explain, and an exit code would turn "explain this" into "re-run until it is quiet". For live benchmark probes, --duration is the shaped publish window; after that rosotacom stops synthetic publishers and waits --drain-s seconds before tearing down shaping, so delayed in-flight messages are not miscounted as loss. Cyclone DDS benchmark sessions keep the default tuned OTA SPDPInterval=30s. On tight rate-limited profiles, this periodic discovery metatraffic can share the shaped link with payload traffic and appear as regular p99/max latency spikes. rosotacom records a cyclonedds_spdp diagnostic in result.json and prints a warning when the probe duration and offered/shaped bandwidth make that effect plausible. Keep the default for end-to-end DDS behavior. For payload-only characterization where discovery bursts would mask the question under test, pass --cyclone-spdp-interval 150s or another longer positive ms/s duration. Making SPDP too frequent improves stale-peer and reconnect detection cadence but contaminates tight-link latency more often; making it too lax quiets short probes but hides discovery/liveliness overhead and delays detection of changed peer state. The living record for this quirk is the SPDP discovery-burst finding. Use benchmark probe when you want a fixed payload/rate under one profile instead of a breakpoint search. It writes the normal result.json plus time-bins.jsonl with per-second loss, delivered Hz, payload bandwidth, and p50/p95 latency/jitter bins; by default it also renders probe-timeseries.png when the optional plotting dependency is installed:

rosotacom benchmark probe --profile cellular-4g-typical --size 18000 --rate-hz 20 --duration 20 --repeats 1
rosotacom benchmark probe --profile cellular-4g-typical --size-pattern 1x20KB+1x0KB --rate-hz 10 --duration 20 --repeats 1
rosotacom benchmark probe --profile cellular-4g-typical --size 18000 --rate-hz 20 --duration 20 --cyclone-spdp-interval 150s
rosotacom benchmark plot time-bins.jsonl --type probe

Use benchmark requirements when you want rosotacom to search for a tight network profile for a stream and quality target. The driver starts from ideal conditions, uses a geometric boundary search for bandwidth, linear boundary searches for latency-like axes, and finishes with an extra refinement pass around the final candidate. By default jitter and loss are searched as a coupled practical tradeoff curve; pass --loss-coupling independent if you want network loss treated as its own axis. For an exact --max-loss 0 target, rosotacom automatically keeps shaped network loss at 0% and searches jitter separately, so the reproducible command encodes the same policy used for the measurement. If the zero-loss boundary is noisy, repeat each candidate and require a pass count; for example, --probe-repeats 10 --probe-min-passes 9 --bad-lossy-count 10 defines a good case as at least nine clean repeats and records bad-case neighbors where all ten repeats lose messages. Add --netem-seed <n> when you want the generated jitter/loss draw to be replayable across repeated boundary searches. For local Docker benchmarks, rosotacom copies the host tc binary into seeded benchmark containers when the container distro tc is too old for netem seed; the host tc must support seed SEED. For a practical good-case reference, --jitter-guard-ratio 0.10 records the tight jitter boundary but uses 10% less jitter for later axes and the final profile; --bandwidth-guard-ratio 0.10 does the symmetric thing for bandwidth by using 10% more bandwidth after finding the lower bound. The chosen profile, target slack, repeat statistics, all probes, and the nearby failing neighbors are written to result.json, requirements.jsonl, and generated-profiles.yaml:

rosotacom benchmark requirements --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-loss 5 --max-latency-ms 250
rosotacom benchmark requirements --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-loss 0 --max-latency-ms 250
rosotacom benchmark requirements --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-loss 0 --max-latency-ms 250 --latency-base-ms 30 --downlink-mode lan --probe-repeats 10 --probe-min-passes 9 --bad-lossy-count 10 --jitter-guard-ratio 0.10 --bandwidth-guard-ratio 0.10 --netem-seed 424242
rosotacom benchmark requirements --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-loss 0 --max-latency-ms 250 --latency-base-ms 30 --downlink-mode lan --axes jitter,bandwidth --bandwidth-high-factor 8 --bandwidth-low-factor 1 --jitter-high-ms 40 --search-iterations 7 --final-refine-iterations 0 --search-rounds 1 --min-duration 20 --min-messages 100 --probe-repeats 1 --bandwidth-probe-repeats 1 --netem-seed 424242 --jitter-guard-ratio 0.20 --bandwidth-guard-ratio 0.20

Use benchmark loss-boundaries when the question is specifically “where does the first ROS 2 loss appear?” It searches bandwidth and jitter as discrete axes: bandwidth reports the lowest loss-free rate and the adjacent lower bad rate at --bandwidth-step; jitter reports the highest fully clean value, the first non-clean neighbor, the lowest fully lossy value, and the mixed zone between them at --jitter-step-ms. Seedless runs are the best empirical estimate for a host/link, a single --netem-seed is useful for replay/debugging one random jitter sequence, and --netem-seeds 101,102,... is the reproducible compromise when you want several replayable jitter draws instead of overfitting one seed:

rosotacom benchmark loss-boundaries --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-latency-ms 250 --latency-base-ms 30 --downlink-mode lan --bandwidth-low 2.8mbit --bandwidth-high 4mbit --bandwidth-step 0.1mbit --jitter-low-ms 0 --jitter-high-ms 40 --jitter-step-ms 1 --min-duration 20 --min-messages 100 --probe-repeats 10
rosotacom benchmark loss-boundaries --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-latency-ms 250 --latency-base-ms 30 --downlink-mode lan --bandwidth-low 2.8mbit --bandwidth-high 4mbit --bandwidth-step 0.1mbit --jitter-low-ms 0 --jitter-high-ms 40 --jitter-step-ms 1 --min-duration 20 --min-messages 100 --probe-repeats 1 --netem-seeds 101,102,103,104,105,106,107,108,109,110

Both requirements and loss-boundaries also accept a bag replay as the load instead of the synthetic sized_publisher stream: pass --target <session> (a two-peer replay session such as 15_remote_assist_anonymized_costmap) and, optionally, --bag <metadata.yaml> for per-topic completeness ground truth. A probe point is then one full replay run under the candidate profile, and the oracle judges the whole contract — every carried topic must clear the loss/latency bound and the --oracle-min-completeness floor — aggregated to a run verdict that names any failing topics. This answers "which network could carry this bag?" rather than "…this synthetic stream?". Because a probe now costs a full bag loop, keep the iteration budget small. See docs/benchmark-bag-as-load.md for the oracle, the local-vs-OTA execution split, and cost guidance:

rosotacom benchmark loss-boundaries --target 15_remote_assist_anonymized_costmap --target-type session --axes bandwidth --bandwidth-low 0.5mbit --bandwidth-high 2mbit --bandwidth-step 0.5mbit --rate-hz 10 --min-duration 20 --min-messages 1 --probe-repeats 1 --good-clean-count 1 --bad-lossy-count 1 --max-latency-ms 1000

Use benchmark ab when the question is a tuning one: "does candidate config B beat baseline config A on the same load and profile?" Each --baseline and --candidate label=… is a whole session config (a directory or a session-definition.yaml) that shares the synthetic a_to_b load and differs only in the pipeline knobs under test (QoS reliability/depth/lifespan, compression, ffmpeg gop/bitrate/crf, … — knobs that keep the delivered topic name, unlike throttle_hz/drop which republish to a renamed topic; see the doc). The driver runs them interleaved with a rotating start over --repeats, then classifies every candidate against the baseline per topic and metric as IMPROVED / WITHIN / REGRESSED — the same verdict language the regression gate uses — with a two-sided tolerance band (--rel-tolerance, --abs-tolerance). The overall verdict fails if any watched metric regressed; the repeat spread (min/median/max) travels with every cell so small-N effects are read honestly. It writes result.json, a markdown table (ab.md), the per-config YAML diffs, and ab.jsonl. See docs/benchmark-ab.md for the config model, interleaving, and the statistical-power note:

rosotacom benchmark ab --profile cellular-4g-degraded --baseline configs/gop30 --candidate gop15=configs/gop15 --size 18000 --rate-hz 20 --duration 20 --repeats 3

Use rosotacom ota-benchmark for the same probes over deployment peers, without having to name a target:

rosotacom ota-benchmark probe --profile cellular-4g-degraded --size 18000 --rate-hz 20 --duration 20 --repeats 1 --peer a=seat_tks --peer b=majestic_tks
rosotacom ota-benchmark capacity --profile cellular-4g-degraded --knob size --low 1 --high 1 --max-loss 30 --max-latency-ms 1000 --duration 10 --repeats 1 --peer a=seat_tks --peer b=majestic_tks
rosotacom ota-benchmark requirements --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-loss 5 --max-latency-ms 250 --peer a=seat_tks --peer b=majestic_tks
rosotacom ota-benchmark loss-boundaries --rate-hz 20 --size 18000 --qos-reliability best_effort --qos-depth 1 --max-latency-ms 250 --latency-base-ms 30 --downlink-mode lan --bandwidth-low 2.8mbit --bandwidth-high 4mbit --bandwidth-step 0.1mbit --jitter-high-ms 40 --jitter-step-ms 1 --probe-repeats 10 --peer a=seat_tks --peer b=majestic_tks

OTA benchmarks default to the benchmark session for the selected genre. Pass --target and --target-type only when you deliberately want to benchmark a project-specific session or scenario instead. Add --interactive to open a tmux operator view with a high-level run window, one fullscreen attachable catmux window per local peer, a network window split into qdisc status and tc/netem command logs, and a results window that prints the final result once. Shaped OTA benchmark profiles need sudo on the remote peers for the generated tc/ip commands. The default --sudo-mode passwordless checks sudo -n and is the right mode for unattended runs. For attended operator runs, pass --sudo-mode askpass to prompt locally once per peer and feed the password to remote sudo -S over SSH stdin without printing or storing it.

Development

For contributor setup, local checks, PR workflow, CI, and releases, see CONTRIBUTING.md. CI behavior is summarized in docs/ci.md, releases in docs/release.md, and issue-driven work tracking in docs/work-items.md.

The test and measurement architecture is recorded in the design RFCs, with measured public effects in the findings ledger; see docs/testing.md for the test taxonomy. The diagnosis-first quality workflow is described in docs/quality-model.md, with owner-facing entrypoints in docs/owner-runbook.md. Reusable maintainer procedures live in docs/playbooks/README.md. Quality rules live in DEVELOPMENT_PRINCIPLES.md.

Choosing the Transport Layer: CycloneDDS or Zenoh

• Use CycloneDDS when all machines share the same ROS_DOMAIN_ID. This is the simplest and most direct configuration.

• Use Zenoh when peers cannot rely on one shared DDS domain, or when you want to split local application nodes and OTA-facing bridge nodes into different ROS 2 domains on each peer. In that split-domain setup this repository generates a standard ROS 2 domain_bridge for /com/... topics locally, while Zenoh carries the /ota/... traffic between peers.

Position in the OTA Communication Landscape

This repository fits into a broader set of ROS-based OTA communication approaches:

• Direct ROS 2 DDS Communication Native DDS (CycloneDDS, Fast DDS), often with custom configuration for constrained or long-range links. The examples in this repository use CycloneDDS to illustrate this approach.

• ROS 2 over Router-like Backbones Some RMW have their own DDS Routers such as eProsima/DDS-Router. Example 4 uses Zenoh to act as a lightweight router layer.

• MQTT-based Approaches Common in cloud/IoT scenarios. Example: ika-rwth-aachen/mqtt_client

• Custom TCP/UDP Teleoperation Stacks Some frameworks implement their manual tcp/udp transportion layers. Example: TUMFTM/teleoperated_driving

How to Cite

If you wish to cite the ROS Communication DevContainer in your work, please use the following citation:

@InProceedings{gontscharow_scalable,
  author    = {Gontscharow, Martin and Doll Jens and Schotschneider, Albert and Bogdoll, Daniel and Orf Stefan and Jestram Johannes and Zofka, Marc and Z\"{o}llner, J. Marius},
  title     = {{Scalable Remote Operation for Autonomous Vehicles: Integration of Cooperative Perception and Open Source Communication}},
  booktitle = {2024 IEEE Intelligent Vehicles Symposium (IV)},
  year      = {2024}
}

Acknowledgements

The research leading to these results was conducted within the project ÖV-LeitmotiF-KI and was funded by the German Federal Ministry for Digital and Transport (BMDV), grant number 45AVF3004A-G. Responsibility for the information and views set out in this publication lies entirely with the authors.