mrg-iot

IoT testbed experiment automation for the Sphere / Merge Testbed platform.

mrg-iot is a command-line tool that drives the entire lifecycle of an IoT experiment on the Merge Testbed — so you don't have to stitch together a dozen mergetb and ssh calls by hand. From a single command it will:

1. Create an experiment and generate its network model for the devices you select.
2. Realize the model (allocate physical hardware).
3. Materialize the realization (bring the allocation online).
4. Create an XDC (eXperiment Development Container) and attach it to the materialization.
5. SSH into the XDC and open SSH tunnels for RTSP video streams (:8554), the SFTPGo file client used for uploads and downloads (:9001), and the experiment's ExperimentControl gRPC channel (:17000).
6. Give you an interactive control loop to drive devices, watch camera feeds in VLC, and pull experiment artifacts.
7. Clean up every resource it created — even on partial failure.

Table of contents

• Requirements
• Installation
• Quick start
• Authentication
• How sessions work
  • Two required arguments, everything else derived
  • One command for the whole lifecycle: run
  • Step by step
  • Commands that take time: run, async-run, and tasks
  • From inside the XDC: cmd ctl
  • Older spellings
• Files
• The spiot_ctl control language
• Command reference
• Input validation rules
• Configuration & on-disk state
• Architecture
• Batch device traffic testing
• Troubleshooting
• Development
• Release process
• License

Requirements

Installation

The supported, production install path is pipx — it puts mrg-iot on your PATH in its own isolated environment:

pipx install mrg-iot

Or, into a regular virtual environment:

python3 -m venv .venv && source .venv/bin/activate
pip install mrg-iot

Verify the install:

mrg-iot --version
mrg-iot --help

Installing VLC (for video)

Quick start

# 1. Authenticate once — credentials are cached under ~/.mrg-iot/session.json
mrg-iot login

# 2. Spin up an experiment end-to-end, interactively
mrg-iot run

# ...or fully scripted, no prompts:
mrg-iot run --non-interactive \
    --project neuiot \
    --devices s-echodot-1 s-googlenest-1 \
    --exp-name myexp \
    --exp-desc "Smart-speaker capture run" \
    --realization realiot \
    --duration 1w \
    --xdc myxdc

Inside the interactive control loop you can issue spiot_ctl commands:

spiot_ctl > exp devices
spiot_ctl > s-echodot-1 click_button
spiot_ctl > exit

To come back to an experiment run left behind, or to drive one from several terminals, use the same prompt on its own:

mrg-iot cmd shell --project neuiot --experiment myexp --realization realiot --xdc myxdc

Authentication

mrg-iot login                                  # prompts for username + password
mrg-iot login -u myuser                        # prompts for the password only
mrg-iot login -u myuser --password-stdin < f   # reads the password from stdin
mrg-iot logout                                 # clears the stored session

Credentials are written to ~/.mrg-iot/session.json with mode 0600. Every authenticated command transparently re-validates the stored token and re-prompts only if the portal rejects it — you rarely need to run login more than once. If the portal simply can't be reached, the stored session is kept rather than discarded, so a brief network problem doesn't cost you a re-login.

You can also supply the username up front and let mrg-iot ask only for the password:

mrg-iot login -u myuser          # prompts for the password only

Unattended authentication (CI, cron, batch scripts)

Two options. Pipe the password in, keeping it out of your shell history and out of the process list:

mrg-iot login -u myuser --password-stdin < ~/.secrets/portal
# or
printf '%s' "$PORTAL_PASSWORD" | mrg-iot login -u myuser --password-stdin

login and logout now exit non-zero on failure, so || exit 1 works.

There is deliberately no --password <pw> flag. On Linux any user can read another process's arguments from /proc/<pid>/cmdline, and on every platform the command is recorded in shell history and echoed into CI logs. --password-stdin avoids all three.

Or supply credentials through the environment, which applies to every subcommand rather than just login — no separate login step and no session.json written:

# Either an existing bearer token...
export MRG_IOT_TOKEN=<token>
export MRG_IOT_USERNAME=<username>   # optional; used for display + SSH key lookup

# ...or a username and password, which are exchanged for a token normally.
export MRG_IOT_USERNAME=<username>
export MRG_IOT_PASSWORD=<password>

mrg-iot cmd run "exp devices"

MRG_IOT_TOKEN takes precedence and skips the login call entirely. MRG_IOT_USERNAME + MRG_IOT_PASSWORD must both be set to take effect, and a rejected password fails immediately rather than retrying (there is nothing to re-prompt). If credentials are needed and none of these are available while stdin is not a terminal, the command exits with an explanatory message rather than a traceback.

Prefer MRG_IOT_TOKEN in shared CI: a token can be revoked with mrg-iot logout, and it keeps the password out of the job configuration.

How sessions work

Everything that talks to a running experiment -- sending commands, watching cameras, moving files -- needs an SSH session to the XDC with tunnels forwarded through it. Each command opens its own session for as long as it runs, then closes it. There is no background daemon and nothing cached between invocations.

Two consequences worth knowing up front:

• Commands that give you something continuous have to keep running. stream open holds its tunnel until you close the viewers. Closing the command closes the thing.
• Sessions are cheap to run in parallel. The local end of every tunnel is an OS-assigned ephemeral port, so you can have stream open in one terminal, cmd shell in another and file download in a third without colliding.

Two required arguments, everything else derived

Which enclave a command targets comes from --project / --experiment / --realization / --xdc. Only two of those are ever required, and the rest are derived, never asked for — so every command in this README runs unattended, in a script or in CI, exactly as written.

Every one of those is still overridable by its flag; the defaults are only what you get when you omit it. So the shortest useful invocation is:

mrg-iot exp setup --devices s-echodot-1   # neuiotadm / realiot / neuiotadmxdc
mrg-iot cmd run "exp devices"             # …and this reaches the same enclave

mrg-iot run is the one exception: it is a guided flow that asks questions. Pass --non-interactive to have it take the defaults above instead.

If a derived name points at something that doesn't exist, you get an error that says which name it tried and which flag overrides it — the command never stops to ask.

One command for the whole lifecycle: run

mrg-iot run provisions an experiment, opens a session, drops you into the command prompt, and tears everything down when you type exit (or send Ctrl-C / SIGTERM). It is the default if you type mrg-iot with no subcommand.

mrg-iot run                       # interactive, prompts for everything
mrg-iot run --simplified          # interactive but skips optional prompts
mrg-iot run --non-interactive ... # fully scripted (see Quick start)

Useful run flags:

Tip: By default run keeps the XDC and experiment on exit, so you can come back to them with mrg-iot cmd shell. Pass --delete-xdc/--delete-exp for a full teardown.

Step by step

The same lifecycle, decomposed -- useful when the experiment should outlive your terminal, or when several people share one enclave.

# Create experiment + realization + materialization + XDC, and attach them
mrg-iot exp setup myexp --project neuiot --devices d1 d2 --xdc myxdc

# Use it, as often as you like, from any terminal
mrg-iot cmd shell     --project neuiot --experiment myexp --realization realiot --xdc myxdc
mrg-iot cmd run       "exp devices" "s-echodot-1 click_button"
mrg-iot stream open
mrg-iot file download --output ./traffic.zip

# Tear it all down
mrg-iot exp teardown myexp --project neuiot --realization realiot --xdc myxdc

cmd shell is the interactive prompt: one foreground session you keep open and type into. cmd run runs a batch and exits.

Commands that take time: run, async-run, and tasks

A device command either answers immediately or is queued — the experiment takes the job, hands back an id, and does the work in the background:

Command 's-echodot-1 click_button' with id: 0, returned:
Command click_button is queued

cmd run finishes the job for you. When a reply says "queued" it sends query wait_result <device> <command> <id> and prints the actual result, so one command in means one result out however long the device takes:

mrg-iot cmd run "s-echodot-1 click_button"
# Click: Button Pusher Operation Successful

cmd async-run doesn't wait. It prints what came straight back and records the job as a task with a handle of its own — t1, t2, … — which you pick up in a later invocation:

mrg-iot cmd async-run "s-echodot-1 hold_button 10"
# Command hold_button is queued
# ✔ t1: s-echodot-1 hold_button (job 3)

mrg-iot cmd tasks          # everything on record, read straight off disk
mrg-iot cmd status t1      # query state: where it has got to
mrg-iot cmd wait t1        # query wait_result: block, then print the result

Handles are ours, not the experiment's. Job ids restart at 0 in every experiment, so they are not unique on your machine and are never what you type; a handle is unique everywhere and carries the project, experiment, realization and XDC its job was started in. That is what makes mrg-iot cmd wait t1 a complete command — no flags, no prompts. A flag that contradicts the record is refused rather than sent, since asking the wrong experiment about job 0 does not fail, it answers about a different job.

cmd tasks opens nothing: no login, no portal call, no session. It deliberately does not show each job's state, which would cost one round trip per row — that is what cmd status is for. Both --project and --experiment narrow the list.

Tasks are forgotten when the experiment they belong to is torn down (exp undeploy, exp teardown, exp delete), because a redeployed experiment starts counting job ids from 0 again and a surviving handle would ask about one job and be answered about another.

cmd run normally records nothing — the job is done by the time it returns. The exception is a wait_result that fails or times out: then the job really is still running, so it is recorded and the handle printed rather than lost.

From inside the XDC: cmd ctl

When you are already inside an XDC, none of the SSH/tunnel machinery is needed -- the experiment's ExperimentControl server is reachable directly over WireGuard. mrg-iot cmd ctl gives you the same prompt with no portal login and no tunnel:

mrg-iot cmd ctl              # default host 192.168.254.1
mrg-iot cmd ctl 10.0.0.5     # override the ExperimentControl host

Each line you type is one RunCommand gRPC call; type exit (or Ctrl-C / Ctrl-D) to quit. Prompt history is saved to ~/.spiot_history.

Removed spellings

The pre-split spellings have been removed. Each was a second name for a subcommand that already existed, routed to the same handler, so there was nothing it could do that its replacement could not:

One caveat when updating a script: cmd run is not quite what send did. send printed "Command click_button is queued" and exited, where cmd run stays for the result. mrg-iot cmd async-run is the closer match to the old behaviour.

mrg-iot connect and mrg-iot disconnect have been removed. They spawned a background daemon that held one session open for every command to share; the same job is now done by mrg-iot cmd shell (a foreground session you keep) or by simply running each command on its own.

Files

The XDC runs an SFTPGo file server that mrg-iot reaches through a tunnel. Only its browser-facing web client is mounted (the REST API is not), so:

# What's there
mrg-iot file list

# Download one file, or the whole `output` directory when you name nothing
mrg-iot file download capture.pcap        # -> ~/Downloads/capture.pcap
mrg-iot file download                     # -> ~/Downloads/<enclave>.zip
mrg-iot file download capture.pcap --output /tmp/run.pcap

# Upload and delete
mrg-iot file upload ./script.py
mrg-iot file remove capture.pcap

# Or browse the file server yourself, in a browser
mrg-iot file client

Names are the file server's own, as file list prints them; all four commands work in its root directory. file download with no name is the exception that takes a directory: output can only come back through the web client's zip endpoint, so it lands as <enclave>.zip.

Every one of these drives the web client programmatically -- parsing the credentials out of the experiment's intro banner and logging in through its form endpoints -- exactly as the exit-time download prompt in run does. No scp/SSH access to the XDC is needed.

The spiot_ctl control language

Whether you're in the interactive run loop or using mrg-iot cmd run, you speak to devices through spiot_ctl. The built-in help command lists everything; the most common verbs:

The three query verbs are the ones you rarely need to type: cmd run sends query wait_result for you when a command comes back queued, and cmd wait / cmd status are those verbs addressed by task handle instead of by <device> <command> <id>. See Commands that take time.

To discover what a specific device supports, ask the experiment once you're connected: <device> commands lists its verbs and <device> <command> help prints one verb's usage line. mrg_iot/data/command_spec.json in this repo documents every command's declared argument types, ranges, and allowed values.

mrg-iot does not check these commands before sending them: the experiment validates its own input, so a missing or out-of-range argument comes back from the experiment rather than from the CLI. (mrg-iot still validates its own arguments — experiment names, durations, and so on.) The one thing it does to a line is rewrite the outdated dev <device> <command> form, which the experiment now answers with "Invalid action type". The client-side checker is kept, commented out at its two call sites — see mrg_iot/core/command_check.py.

Command reference

mrg-iot exposes resource-CRUD subcommands so you can manage individual testbed objects without running the full flow. Add --help to any subcommand for its exact flags, and --debug/-d to any command for verbose logging.

Experiments

The lifecycle verbs take the experiment name as a positional NAME (defaulting to your username), so the whole flow can be driven one named experiment at a time:

mrg-iot exp create   myexp --devices d1,d2   # experiment + model; nothing allocated yet
mrg-iot exp deploy   myexp                   # realize -> materialize -> xdc -> attach
mrg-iot exp status   myexp                   # created / realized / materialized / deployed
mrg-iot exp undeploy myexp                   # detach -> delete xdc -> delete mtz + rz
mrg-iot exp delete   myexp                   # remove the experiment record

The verbs pair up, and the two halves have a one-command spelling each:

mrg-iot exp setup    myexp --devices d1,d2   # create + deploy, in one command
mrg-iot exp teardown myexp                   # undeploy + delete, its inverse

teardown runs every step of undeploy before deleting the experiment, so it leaves nothing allocated. An xdc or realization that is already gone is reported as skipped, not as a failure — exp undeploy && exp teardown still exits 0.

--devices/-dev takes either spelling — -dev d1 d2 or -dev d1,d2 — and duplicates are dropped. Because the flag is greedy, NAME has to come before it: exp create -dev d1 d2 myexp reads as three devices and no name.

deploy, setup, undeploy and teardown also take --realization/-r and --xdc/-x; deploy and setup add --duration/-dur and --xdc-duration/-xdur; create and setup take --description/-desc and --network/-net. All of them take --project/-p. Anything omitted falls back to the usual defaults (realiot, <experiment>xdc, 1w, mrg-iot-net) — see Two required arguments.

Inspection, and the older flag spellings, still work:

mrg-iot exp list [--project p] [--all]                      # yours; --all for everyone's
mrg-iot exp create --project p --name n --devices d1,d2 [--description d]
mrg-iot exp delete --project p --name n

Realizations (alias: rz)

mrg-iot realization list   --project p --experiment e
mrg-iot realization status --project p --experiment e --realization r
mrg-iot realization create --project p --experiment e --devices d1 d2 \
        [--name r] [--network net] [--duration 1w]          # build model, compile, push, realize
mrg-iot realization delete --project p --experiment e --realization r

Materializations (alias: mtz)

mrg-iot materialization list          --project p --experiment e
mrg-iot materialization status        --project p --experiment e --realization r
mrg-iot materialization create        --project p --experiment e --realization r
mrg-iot materialization delete        --project p --experiment e --realization r

XDCs

mrg-iot xdc list [--project p]
mrg-iot xdc status --project p --name n
mrg-iot xdc create --project p [--name n] [--duration 1w] [--no-wait]
mrg-iot xdc attach --project p --name n --experiment e --realization r
mrg-iot xdc detach --project p --name n --experiment e --realization r
mrg-iot xdc delete --project p --name n

Devices

mrg-iot devices list [--project p]      # devices free to use (omit --project for all)

Only unallocated devices are listed -- including hiding the ones your own experiment holds. For the devices inside a running experiment, ask the experiment: mrg-iot cmd run "exp devices".

Talking to a running experiment

Each of these opens its own session; all take --project / --experiment / --realization / --xdc, and prompt for whatever you omit.

mrg-iot cmd run "<cmd>" ["<cmd>" ...]   # run a batch, waiting for queued commands
mrg-iot cmd async-run "<cmd>" [...]     # don't wait; record each queued job as a task
mrg-iot cmd tasks [--project p]         # the jobs on record (local; opens nothing)
mrg-iot cmd wait TASK                   # block until that job finishes, print its result
mrg-iot cmd status TASK                 # just its state
mrg-iot cmd shell                       # interactive prompt
mrg-iot cmd ctl [host]                  # same prompt, from inside the XDC

mrg-iot stream open                     # VLC viewers; runs until they close
mrg-iot stream list                     # name the cameras
mrg-iot stream url                      # RTSP URLs; keeps forwarding until Ctrl+C

mrg-iot file list                       # what's in the experiment's directory
mrg-iot file download [name] [-o path]  # a file, or the whole output dir as a zip
mrg-iot file upload <path>              # put a local file in the experiment
mrg-iot file remove <name>              # delete one
mrg-iot file client                     # the web client, in a browser, until Ctrl+C

Typical cleanup sequence

Usually just mrg-iot exp teardown myexp. The long form, when you need to drive one resource at a time:

mrg-iot xdc detach --project neuiot --name myxdc --experiment myexp --realization realiot
mrg-iot xdc delete --project neuiot --name myxdc
mrg-iot materialization delete --project neuiot --experiment myexp --realization realiot
mrg-iot realization delete --project neuiot --experiment myexp --realization realiot
mrg-iot exp delete --project neuiot --name myexp

Input validation rules

Inputs are validated up front so bad arguments fail fast with a clear [mrg-iot] Invalid <label>: … message (exit code 2) rather than deep inside a portal call:

• Names (exp-name, realization, xdc, network): must match ^[a-z][a-z0-9]*$ — start with a lowercase letter, lowercase letters and digits only, max 32 chars.
• Description: letters, digits, spaces, commas, periods, hyphens; max 256 chars.
• Duration: minimum 4 days. Accepts forms like 1w, 4d, 1w2d3h, 1 week. Anything under 4 days is rejected.

Configuration & on-disk state

Everything user-scoped lives under ~/.mrg-iot/:

That is nearly the whole of it. There is no cached connection state: which enclave a command targets comes from its flags or from the interactive pickers, every time. tasks.json is not an exception — it remembers what a job is (device, command, job id, and the enclave it ran in), never a live connection. connection.json, connection.sock and daemon.log belonged to the removed daemon and are safe to delete if an older version left them behind.

Environment variables that affect behaviour:

--debug raises the log level; logs are always written to ~/.mrg-iot/debug.log (appended across runs, rotated at 5 MB, 3 kept). Attach that file and mrg-iot --version to any bug report.

Endpoints & known projects

• gRPC portal: grpc.sphere-testbed.net:443 (via mergetbapi==1.3.41)
• SSH jump host: jump.sphere-testbed.net:2022
• Tunneled ports: 8554 (RTSP), 9001 (SFTPGo file client — uploads/downloads), 17000 (ExperimentControl). Every local port is OS-assigned.
• Known projects: neuiot, iotbeta, iotdev (the hardcoded PROJECTS list in mrg_iot/constants.py; a project outside it is not listed)

Architecture

Three tiers, with a dependency rule enforced by tests/test_import_layering.py:

• core/ — what more than one command needs: reaching an experiment (SSH, tunnels, session), naming which one to reach, and preparing a command line before sending it. Never imports a command group.
• commands/<group>/ — one package per subcommand family. Each owns its own argparse subtree and handlers, and never imports another group.
• views/ — the only place that writes to the terminal. Nothing outside it calls print() or touches rich directly.

mrg_iot/
├── cli.py                  # a registry: GROUPS + main() dispatcher
├── constants.py            # pure data: paths, endpoints, model template, help text
├── core/
│   ├── controller.py       # @require_login
│   ├── enclave.py          # interactive pickers + readiness checks
│   ├── ssh.py              # XDC SSH connect via the jump host
│   ├── tunnel.py           # Paramiko port forwarding (ephemeral local ports)
│   ├── command_check.py    # legacy-syntax rewrite; argument checking (disabled)
│   ├── auth.py             # credential precedence + login/logout
│   ├── api/                # portal gRPC client + retry wrapper
│   └── session/            # model, scope, channel, transport, checking, intro
├── commands/
│   ├── cmd/                # run | async-run | wait | status | tasks | shell | ctl
│   │                       #   (+ jobs.py: queued replies; the shared REPL loop)
│   ├── devices/            # list
│   ├── exp/                # exp | realization | materialization | xdc
│   ├── file/               # download | upload | list | remove (SFTPGo web client)
│   ├── session/            # login | logout
│   ├── stream/             # open | list | url  (+ the VLC viewer handling)
│   └── run.py              # the one-command lifecycle
├── models/                 # dataclasses + on-disk state
├── views/                  # all user-facing output
└── utils/                  # duration parsing, file helpers

The run workflow (commands/run.py, composing commands/exp/provision.py):

Login → Select Project → Select Devices → Create Experiment →
Build+Compile+Push Model → Realize → Materialize → Create/Attach XDC →
SSH + tunnels + ExperimentControl channel → command loop → Cleanup

Progress is recorded in an ExpState dataclass as each resource is created, so cleanup tears down exactly what exists even when provisioning fails partway.

Sessions. core.session.session_scope is the single way to open one, for every command. It is a context manager rather than an open/close pair specifically so a Ctrl-C partway through building the tunnels cannot leak the SSH client. Each local tunnel port is assigned by the OS, and URLs pointing at forwarded services (RTSP, the file server) are built from the live session's port map — there is no fixed port to hardcode.

Batch device traffic testing

run_device_tests.sh drives mrg-iot run over a whole device inventory and reports, per device, whether packets from that device actually appeared in the experiment's capture — PASS / FAIL / SKIP plus a summary CSV.

EXP_NAME=bulbs ./run_device_tests.sh --repo bulbs_repo.yaml -n 20

Full operating guide, including the device repo format, batched mode, reruns of just the failures, and the --jobs caveat: docs/RUN_DEVICE_TESTS.md.

Troubleshooting

SSL / TLS error talking to the portal (common on macOS):

export SSL_CERT_FILE="$(python -m certifi)"
export REQUESTS_CA_BUNDLE="$SSL_CERT_FILE"

stream open does nothing / errors: Make sure the vlc binary is installed and on your PATH (vlc --version). pip/pipx cannot install VLC. If you would rather use another player, mrg-iot stream url prints the RTSP addresses and keeps forwarding them.

A file command can't find your file: Names are the file server's, not your shell's — no globbing and no paths. Run mrg-iot file list and use the name it prints.

VLC connects but shows nothing: Check for an editor forwarding the same port. VS Code's automatic port forwarding binds 127.0.0.1:<port> and shadows the tunnel for anything connecting to localhost — remove the entry from its Ports panel, or set "remote.autoForwardPorts": false.

A command hangs or times out: Re-run with --debug and inspect ~/.mrg-iot/debug.log for the underlying gRPC/SSH error.

Resources left behind after a crash: Use the cleanup sequence to remove the XDC, materialization, realization, and experiment individually.

Development

Known gaps, open defects and the prioritised roadmap live in docs/GAPS.md — read that before picking up work, and update it when you close something.

git clone https://gitlab.com/sphere-neu/mrg-iot.git
cd mrg-iot
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Tests, type-checking, linting, formatting:

pytest tests/ -v                              # asyncio_mode=auto; no extra flags
pytest tests/test_commands_cmd.py -v          # a single file
mypy mrg_iot
pylint mrg_iot
black mrg_iot tests && isort mrg_iot tests

The dev extras (black, pylint, isort, etc.) require Python 3.10+. On a 3.9 environment, install runtime deps only (pip install -e .).

Pinned dependency lockfiles live under requirements/ (requirements/prod.txt for runtime deps, requirements/dev.txt for the lint/format/type-check tools) if you'd rather install from those than from pyproject.toml's extras:

pip install -r requirements/prod.txt   # runtime only, Python 3.9+
pip install -r requirements/dev.txt    # runtime + dev tools, Python 3.10+

Release process

CI (.gitlab-ci.yml) runs pytest on every push, builds an sdist + wheel on success, and publishes to PyPI on vX.Y.Z tags via OIDC trusted publishing.

The git tag must equal v$(cat mrg_version) — the publish job aborts otherwise. To cut a release:

# bump the version
echo "1.3.2" > mrg_version
git commit -am "increased version to 1.3.2"
git tag v1.3.2
git push && git push --tags

License

MIT © Sphere / NEU IoT Testbed contributors