mrg-iot
IoT testbed experiment automation for the Sphere / Merge Testbed platform.
mrg-iot is a command-line tool that drives the entire lifecycle of an IoT
experiment on the Merge Testbed — so you don't have to stitch together a dozen
mergetb and ssh calls by hand. From a single command it will:
- Create an experiment and generate its network model for the devices you select.
- Realize the model (allocate physical hardware).
- Materialize the realization (bring the allocation online).
- Create an XDC (eXperiment Development Container) and attach it to the materialization.
- SSH into the XDC and open SSH tunnels for RTSP video streams (
:8554), the SFTPGo file client used for uploads and downloads (:9001), and the experiment'sExperimentControlgRPC channel (:17000). - Give you an interactive control loop to drive devices, watch camera feeds in VLC, and pull experiment artifacts.
- Clean up every resource it created — even on partial failure.
Table of contents
- Requirements
- Installation
- Quick start
- Authentication
- How sessions work
- Files
- The
spiot_ctlcontrol language - Command reference
- Output: stdout vs stderr
- Input validation rules
- Configuration & on-disk state
- Architecture
- Troubleshooting
- Development
- Release process
- License
Requirements
| Requirement | Notes |
|---|---|
| Python 3.9+ | 3.9, 3.10, 3.11, and 3.12 are supported and CI-tested. |
| A Merge Testbed account | With access to at least one IoT project (neuiot, iotbeta, or iotdev). |
| Network access | To the gRPC portal (grpc.sphere-testbed.net:443) and the SSH jump host (jump.sphere-testbed.net:2022). |
| VLC | Only needed for stream open / RTSP playback. pip cannot install it — install it from your OS package manager. The vlc binary must be on your PATH. |
Installation
The supported, production install path is pipx — it
puts mrg-iot on your PATH in its own isolated environment:
pipx install mrg-iot
Or, into a regular virtual environment:
python3 -m venv .venv && source .venv/bin/activate
pip install mrg-iot
Verify the install:
mrg-iot --version
mrg-iot --help
Installing VLC (for video)
| Platform | Command |
|---|---|
| macOS | brew install --cask vlc |
| Debian/Ubuntu | sudo apt-get install vlc |
| Fedora | sudo dnf install vlc |
| Windows | Download from videolan.org |
Quick start
# 1. Authenticate once — credentials are cached under ~/.mrg-iot/session.json
mrg-iot login
# 2. Spin up an experiment end-to-end, interactively
mrg-iot run
# ...or fully scripted, no prompts:
mrg-iot run --non-interactive \
--project neuiot \
--devices s-echodot-1 s-googlenest-1 \
--exp-name myexp \
--exp-desc "Smart-speaker capture run" \
--realization realiot \
--duration 1w \
--xdc myxdc
Inside the interactive control loop you can issue spiot_ctl commands:
spiot_ctl > exp devices
spiot_ctl > s-echodot-1 click_button
spiot_ctl > exit
To come back to an experiment run left behind, or to drive one from several
terminals, use the same prompt on its own:
mrg-iot cmd shell --project neuiot --experiment myexp --realization realiot --xdc myxdc
Authentication
mrg-iot login # prompts for username + password
mrg-iot login -u myuser # prompts for the password only
mrg-iot login -u myuser --password-stdin < f # reads the password from stdin
mrg-iot logout # clears the stored session
Credentials are written to ~/.mrg-iot/session.json with mode 0600. Every
authenticated command transparently re-validates the stored token and
re-prompts only if the portal rejects it — you rarely need to run login
more than once. If the portal simply can't be reached, the stored session is
kept rather than discarded, so a brief network problem doesn't cost you a
re-login.
You can also supply the username up front and let mrg-iot ask only for the
password:
mrg-iot login -u myuser # prompts for the password only
Unattended authentication (CI, cron, batch scripts)
Two options. Pipe the password in, keeping it out of your shell history and out of the process list:
mrg-iot login -u myuser --password-stdin < ~/.secrets/portal
# or
printf '%s' "$PORTAL_PASSWORD" | mrg-iot login -u myuser --password-stdin
login and logout now exit non-zero on failure, so || exit 1 works.
There is deliberately no
--password <pw>flag. On Linux any user can read another process's arguments from/proc/<pid>/cmdline, and on every platform the command is recorded in shell history and echoed into CI logs.--password-stdinavoids all three.
Or supply credentials through the environment, which applies to every
subcommand rather than just login — no separate login step and no
session.json written:
# Either an existing bearer token...
export MRG_IOT_TOKEN=<token>
export MRG_IOT_USERNAME=<username> # optional; used for display + SSH key lookup
# ...or a username and password, which are exchanged for a token normally.
export MRG_IOT_USERNAME=<username>
export MRG_IOT_PASSWORD=<password>
mrg-iot cmd run "exp devices"
MRG_IOT_TOKEN takes precedence and skips the login call entirely.
MRG_IOT_USERNAME + MRG_IOT_PASSWORD must both be set to take effect, and a
rejected password fails immediately rather than retrying (there is nothing to
re-prompt). If credentials are needed and none of these are available while
stdin is not a terminal, the command exits with an explanatory message rather
than a traceback.
Prefer
MRG_IOT_TOKENin shared CI: a token can be revoked withmrg-iot logout, and it keeps the password out of the job configuration.
How sessions work
Everything that talks to a running experiment -- sending commands, watching cameras, moving files -- needs an SSH session to the XDC with tunnels forwarded through it. Each command opens its own session for as long as it runs, then closes it. There is no background daemon and nothing cached between invocations.
Two consequences worth knowing up front:
- Commands that give you something continuous have to keep running.
stream openholds its tunnel until you close the viewers. Closing the command closes the thing. - Sessions are cheap to run in parallel. The local end of every tunnel is
an OS-assigned ephemeral port, so you can have
stream openin one terminal,cmd shellin another andfile downloadin a third without colliding.
Two required arguments, everything else derived
Which enclave a command targets comes from --project / --experiment / --realization / --xdc. Only two of those are ever required, and the rest are
derived, never asked for — so every command in this README runs unattended,
in a script or in CI, exactly as written.
| Argument | Rule |
|---|---|
--project, -p |
Required only if you belong to more than one project. With one, it's used automatically; with several and no flag, the error lists them. |
--devices, -d |
Required only on the commands that build a device model: exp create, exp setup, run. |
--experiment, -e |
Defaults to your username — neuiotadm gets the experiment neuiotadm. |
--xdc, -x |
Defaults to <experiment>xdc — so exp setup myexp provisions myexpxdc, and -e myexp alone is enough to reach it later. |
--realization, -r |
Defaults to realiot. |
--duration, -dur / --xdc-duration, -xdur |
Default to 1w (minimum 4 days). |
--network, -net |
Defaults to mrg-iot-net. |
--description, -desc |
Defaults to Experiment created with mrg-iot by <username>. |
Every one of those is still overridable by its flag; the defaults are only what you get when you omit it. So the shortest useful invocation is:
mrg-iot exp setup --devices s-echodot-1 # neuiotadm / realiot / neuiotadmxdc
mrg-iot cmd run "exp devices" # …and this reaches the same enclave
mrg-iot run is the one exception: it is a guided flow that asks questions.
Pass --non-interactive to have it take the defaults above instead.
If a derived name points at something that doesn't exist, you get an error that says which name it tried and which flag overrides it — the command never stops to ask.
One command for the whole lifecycle: run
mrg-iot run provisions an experiment, opens a session, drops you into the
command prompt, and tears everything down when you type exit (or send Ctrl-C
/ SIGTERM). It has to be named: bare mrg-iot prints usage.
mrg-iot run # interactive, prompts for everything
mrg-iot run --simplified # interactive but skips optional prompts
mrg-iot run --non-interactive ... # fully scripted (see Quick start)
Useful run flags:
| Flag | Purpose |
|---|---|
--non-interactive, -n |
No prompts; all required args must be supplied. |
--simplified, -s |
Interactive, but uses sensible defaults for optional steps. |
--commands-file, -cf <path> |
Run a newline-delimited file of commands non-interactively. |
--download-files, -df |
Download the experiment archive on exit. |
--delete-xdc, -dx |
Delete the XDC during cleanup (default: keep). |
--delete-exp, -de |
Delete the experiment during cleanup (default: keep). |
--network, -net <name> |
Model network name (default: mrg-iot-net). |
Tip: By default
runkeeps the XDC and experiment on exit, so you can come back to them withmrg-iot cmd shell. Pass--delete-xdc/--delete-expfor a full teardown.
Step by step
The same lifecycle, decomposed -- useful when the experiment should outlive your terminal, or when several people share one enclave.
# Create experiment + realization + materialization + XDC, and attach them
mrg-iot exp setup myexp --project neuiot --devices d1 d2 --xdc myxdc
# Use it, as often as you like, from any terminal
mrg-iot cmd shell --project neuiot --experiment myexp --realization realiot --xdc myxdc
mrg-iot cmd run "exp devices" "s-echodot-1 click_button"
mrg-iot stream open
mrg-iot file download --output ./traffic.zip
# Tear it all down
mrg-iot exp teardown myexp --project neuiot --realization realiot --xdc myxdc
cmd shell is the interactive prompt: one foreground session you keep open and
type into. cmd run runs a batch and exits.
Commands that take time: run, async-run, and tasks
A device command either answers immediately or is queued — the experiment takes the job, hands back an id, and does the work in the background:
Command 's-echodot-1 click_button' with id: 0, returned:
Command click_button is queued
cmd run finishes the job for you. When a reply says "queued" it sends
query wait_result <device> <command> <id> and prints the actual result, so one
command in means one result out however long the device takes:
mrg-iot cmd run "s-echodot-1 click_button"
# Click: Button Pusher Operation Successful
cmd async-run doesn't wait. It prints what came straight back and records the
job as a task with a handle of its own — t1, t2, … — which you pick up
in a later invocation:
mrg-iot cmd async-run "s-echodot-1 hold_button 10"
# Command hold_button is queued
# ✔ t1: s-echodot-1 hold_button (job 3)
mrg-iot cmd tasks # everything on record, read straight off disk
mrg-iot cmd status t1 # query state: where it has got to
mrg-iot cmd wait t1 # query wait_result: block, then print the result
Handles are ours, not the experiment's. Job ids restart at 0 in every
experiment, so they are not unique on your machine and are never what you type;
a handle is unique everywhere and carries the project, experiment, realization
and XDC its job was started in. That is what makes mrg-iot cmd wait t1 a
complete command — no flags, no prompts. A flag that contradicts the record is
refused rather than sent, since asking the wrong experiment about job 0 does not
fail, it answers about a different job.
cmd tasks opens nothing: no login, no portal call, no session. It deliberately
does not show each job's state, which would cost one round trip per row — that
is what cmd status is for. Both --project and --experiment narrow the list.
Tasks are forgotten when the experiment they belong to is torn down
(exp undeploy, exp teardown, exp delete), because a
redeployed experiment
starts counting job ids from 0 again and a surviving handle would ask about one
job and be answered about another.
cmd run normally records nothing — the job is done by the time it returns. The
exception is a wait_result that fails or times out: then the job really is
still running, so it is recorded and the handle printed rather than lost.
From inside the XDC: cmd ctl
When you are already inside an XDC, none of the SSH/tunnel machinery is
needed -- the experiment's ExperimentControl server is reachable directly over
WireGuard. mrg-iot cmd ctl gives you the same prompt with no portal login and
no tunnel:
mrg-iot cmd ctl # default host 192.168.254.1
mrg-iot cmd ctl 10.0.0.5 # override the ExperimentControl host
Each line you type is one RunCommand gRPC call; type exit (or Ctrl-C /
Ctrl-D) to quit. Prompt history is saved to ~/.spiot_history.
Removed spellings
The pre-split spellings have been removed. Each was a second name for a subcommand that already existed, routed to the same handler, so there was nothing it could do that its replacement could not:
| Removed | Use instead |
|---|---|
mrg-iot send ... |
mrg-iot cmd run ... |
mrg-iot cmd send ... |
mrg-iot cmd run ... |
mrg-iot ctl |
mrg-iot cmd ctl |
mrg-iot show video |
mrg-iot stream open |
mrg-iot download traffic |
mrg-iot file download |
One caveat when updating a script: cmd run is not quite what send did.
send printed "Command click_button is queued" and exited, where cmd run
stays for the result. mrg-iot cmd async-run is the closer match to the
old behaviour.
mrg-iot connect and mrg-iot disconnect have been removed. They spawned a
background daemon that held one session open for every command to share; the
same job is now done by mrg-iot cmd shell (a foreground session you keep) or
by simply running each command on its own.
Files
The XDC runs an SFTPGo file server that mrg-iot reaches through a tunnel.
Only its browser-facing web client is mounted (the REST API is not), so:
# What's there -- the root, or any directory in it
mrg-iot file list
mrg-iot file list output
# Download one file, or the whole `output` directory when you name nothing.
# A download is named for the experiment it came from and where in the share
# it sat, with the remote path flattened onto dots.
mrg-iot file download capture.pcap # -> ~/Downloads/myexp.capture.pcap
mrg-iot file download output/capture.pcap # -> ~/Downloads/myexp.output.capture.pcap
mrg-iot file download traces/capture.pcap # -> ~/Downloads/myexp.traces.capture.pcap
mrg-iot file download # -> ~/Downloads/myexp.output.zip
mrg-iot file download capture.pcap --output /tmp/run.pcap
# Upload and delete
mrg-iot file upload ./script.py
mrg-iot file remove capture.pcap
# Or browse the file server yourself, in a browser
mrg-iot file client
Names are the file server's own, as file list prints them. list and
download also take a path into a subdirectory (output/capture.pcap).
What a download is called locally is <experiment>.<remote path>, with every
/ turned into a .. The remote layout is never mirrored -- joining it whole
would silently create a ~/Downloads/output/ on the way past, which no other
download tool does -- but flattening it onto the name keeps two facts a bare
basename lost: which experiment the file came from, and which directory it sat
in, so output/capture.pcap and traces/capture.pcap no longer overwrite each
other. An experiment name can't contain a . (names are ^[a-z][a-z0-9]*$), so
the first segment is always unambiguous. Pass --output/-o for a name of your
own.
upload and remove work in the root only -- uploads always go to uploads/,
since the root is read-only for the experiment's SFTPGo user. file download
with no name is the one that takes a directory: output can only come back
through the web client's zip endpoint, so it lands as <experiment>.output.zip.
Every one of these drives the web client programmatically -- parsing the
credentials out of the experiment's intro banner and logging in through its
form endpoints -- exactly as the exit-time download prompt in run does. No
scp/SSH access to the XDC is needed.
The spiot_ctl control language
Whether you're in the interactive run loop or using mrg-iot cmd run, you speak
to devices through spiot_ctl. The built-in help command lists everything;
the most common verbs:
| Command | Description |
|---|---|
help |
Show all available commands. |
exp devices |
List the devices in the experiment. |
exp cred <device> |
Get a device's credentials (e.g. exp cred s-echodot-1). |
exp sleep <seconds> |
Pause for N seconds. |
exp read <filepath> |
Read a command file on the XDC and run every line in it. |
<device> help / <device> commands |
Show the commands a device supports. |
<device> <command> help |
Detailed help for one device command. |
| ` [-s <id | result |
query state <device> <command> <id> |
Check the state of a running command. |
query get_result <device> <command> <id> |
Fetch the result of a finished command. |
query wait_result <device> <command> <id> |
Block until a command finishes, then return its result. |
exit |
Tear down spiot_ctl, optionally download files, and clean up. |
The three query verbs are the ones you rarely need to type: cmd run sends
query wait_result for you when a command comes back queued, and cmd wait /
cmd status are those verbs addressed by task handle instead of by
<device> <command> <id>. See
Commands that take time.
To discover what a specific device supports, ask the experiment once you're
connected: <device> commands lists its verbs and <device> <command> help
prints one verb's usage line. mrg_iot/data/command_spec.json in this repo
documents every command's declared argument types, ranges, and allowed values.
mrg-iot does not check these commands before sending them: the experiment
validates its own input, so a missing or out-of-range argument comes back from
the experiment rather than from the CLI. (mrg-iot still validates its own
arguments — experiment names, durations, and so on.) The one thing it does to a
line is rewrite the outdated dev <device> <command> form, which the experiment
now answers with "Invalid action type". The client-side checker is kept, commented
out at its two call sites — see mrg_iot/core/command_check.py.
Command reference
mrg-iot exposes resource-CRUD subcommands so you can manage individual
testbed objects without running the full flow. Add --help to any subcommand
for its exact flags. Set MRG_IOT_TRACE=1 in the environment for verbose
logging (there is no --debug flag).
Experiments
The lifecycle verbs take the experiment name as a positional NAME
(defaulting to your username), so the whole flow can be driven one named
experiment at a time:
mrg-iot exp create myexp --devices d1,d2 # experiment + model; nothing allocated yet
mrg-iot exp deploy myexp # realize -> materialize -> xdc -> attach
mrg-iot exp status myexp # created / realized / materialized / deployed
mrg-iot exp undeploy myexp # detach -> delete xdc -> delete mtz + rz
mrg-iot exp delete myexp # remove the experiment record
The verbs pair up, and the two halves have a one-command spelling each:
mrg-iot exp setup myexp --devices d1,d2 # create + deploy, in one command
mrg-iot exp teardown myexp # undeploy + delete, its inverse
teardown runs every step of undeploy before deleting the experiment, so it
leaves nothing allocated. An xdc or realization that is already gone is reported
as skipped, not as a failure — exp undeploy && exp teardown still exits 0.
--devices/-d takes either spelling — -d d1 d2 or -d d1,d2 — and
duplicates are dropped. Because the flag is greedy, NAME has to come before
it: exp create -d d1 d2 myexp reads as three devices and no name.
deploy, setup, undeploy and teardown also take --realization/-r and
--xdc/-x; deploy and setup add --duration/-dur and
--xdc-duration/-xdur; create and setup take --description/-desc and
--network/-net. All of them take --project/-p.
Anything omitted falls back to the usual defaults (realiot,
<experiment>xdc, 1w, mrg-iot-net) — see
Two required arguments.
Inspection, and the older flag spellings, still work:
mrg-iot exp list [--project p] [--all] # yours; --all for everyone's
mrg-iot exp create --project p --name n --devices d1,d2 [--description d]
mrg-iot exp delete --project p --name n
Devices
mrg-iot devices list # the devices in your experiment
mrg-iot devices list --experiment e # ...in another one
mrg-iot devices info # describe every device in it
mrg-iot devices info -d s-echodot-1 s-veldslight-3 # ...or just these
mrg-iot devices list --available [--project p] # devices free to *use* (portal)
list and info describe a running experiment, so both take the usual
--project / --experiment / --realization / --xdc and derive whatever you omit
(experiment <username>, xdc <experiment>xdc, realization realiot). Both
read the experiment's GetIntro banner and nothing else -- it is the only place
it publishes its device list and its per-device facts, and it arrives with the
session handshake, so neither command costs a device command.
list names the devices the experiment is about: s-/m- names, not the
h- handler cameras (those are stream list). info prints what the banner
says about each one:
Device: s-echodot-1
VLAN: 3001
MAC: 9c:c8:e9:cc:27:29
Device IP: 172.30.1.3
Gateway: 172.30.1.1
Camera: h-camera-2
Every field the banner publishes is shown, so one it grows later appears here with no new release. The camera is paired by position -- the banner lists cameras in device order and gives no mapping -- and only when there is exactly one camera per device; any other count leaves the row out rather than guessing.
--available is the other question -- what is free to put in an experiment --
and is the one path that opens no session. Only unallocated devices are
listed, including hiding the ones your own experiment holds.
Talking to a running experiment
Each of these opens its own session; all take
--project / --experiment / --realization / --xdc and derive whatever you
omit (experiment <username>, xdc <experiment>xdc, realization realiot).
None of them prompts, so every line below is scriptable exactly as written.
mrg-iot cmd run "<cmd>" ["<cmd>" ...] # run a batch, waiting for queued commands
mrg-iot cmd async-run "<cmd>" [...] # don't wait; record each queued job as a task
mrg-iot cmd tasks [--project p] # the jobs on record (local; opens nothing)
mrg-iot cmd wait TASK # block until that job finishes, print its result
mrg-iot cmd status TASK # just its state
mrg-iot cmd shell # interactive prompt
mrg-iot cmd ctl [host] # same prompt, from inside the XDC
mrg-iot stream open # VLC viewers; runs until they close
mrg-iot stream list # name the cameras
mrg-iot stream url # RTSP URLs; keeps forwarding until Ctrl+C
mrg-iot file list [path] # what's in the experiment's directory
mrg-iot file download [name] [-o path] # a file, or the whole output dir as a zip
mrg-iot file upload <path> # put a local file in the experiment
mrg-iot file remove <name> # delete one
mrg-iot file client # the web client, in a browser, until Ctrl+C
Typical cleanup sequence
mrg-iot exp teardown myexp is the whole of it — it detaches and deletes the
XDC, releases the hardware, and deletes the experiment record, reporting each
resource it removed and skipping anything already gone.
mrg-iot exp teardown myexp --project neuiot # release everything
mrg-iot exp undeploy myexp --project neuiot # release the hardware, keep the experiment
Output: stdout vs stderr
mrg-iot follows the usual Unix split, so its commands compose with other
tools:
- stdout carries the primary output and nothing else — tables, key/value blocks, and verbatim output from the experiment or the portal.
- stderr carries everything else —
✔/✖/▲lines, progress narration, banners, prompts and spinners.
So the data is what a redirect collects, and the commentary still reaches your terminal:
# Just the URLs, straight into a player.
mrg-iot stream url 2>/dev/null | xargs vlc
# Just the table, with the ✔/▲ lines still on screen.
mrg-iot exp list > experiments.txt
# A failure is reported on stderr and never written into the data file.
mrg-iot exp status doesnotexist > out.txt || echo "failed; out.txt is empty"
# Second column of your experiments, skipping the header and rule.
mrg-iot exp list 2>/dev/null | awk 'NR>2 {print $2}'
Two things deliberately stay on stdout because moving them would break
something: the cmd shell / cmd ctl prompt (readline needs to own the prompt
string for arrow-key editing and history — the same reason python, psql and
sqlite3 do it), and --help, which argparse writes there by convention.
Password prompts appear on /dev/tty and are on neither stream.
Input validation rules
Inputs are validated up front so bad arguments fail fast with a clear
[mrg-iot] Invalid <label>: … message (exit code 2) rather than deep inside a
portal call:
- Names (
exp-name,realization,xdc,network): must match^[a-z][a-z0-9]*$— start with a lowercase letter, lowercase letters and digits only, max 32 chars. - Description: letters, digits, spaces, commas, periods, hyphens; max 256 chars.
- Duration: minimum 4 days. Accepts forms like
1w,4d,1w2d3h,1 week. Anything under 4 days is rejected.
Configuration & on-disk state
Everything user-scoped lives under ~/.mrg-iot/:
| File | Purpose |
|---|---|
session.json |
{"username", "token"} — your cached login (mode 0600). |
debug.log |
Verbose CLI log, appended across runs and rotated at 5 MB (3 kept). Attach this to bug reports. |
tasks.json |
Background device jobs left running by cmd async-run, so cmd wait/cmd status/cmd tasks can name one in a later invocation (mode 0600). Cleared per experiment when it is undeployed or deleted. |
That is nearly the whole of it. There is no cached connection state: which
enclave a command targets comes from its flags or from the interactive pickers,
every time. tasks.json is not an exception — it remembers what a job is
(device, command, job id, and the enclave it ran in), never a live connection. connection.json, connection.sock and daemon.log belonged to the
removed daemon and are safe to delete if an older version left them behind.
Environment variables that affect behaviour:
| Variable | Effect |
|---|---|
MRG_IOT_TOKEN |
Use this bearer token directly; skips login entirely. |
MRG_IOT_USERNAME |
Username for unattended login, and the account name shown/used for SSH key lookup. |
MRG_IOT_PASSWORD |
Password for unattended login (requires MRG_IOT_USERNAME). |
MRG_IOT_TRACE |
1/true/yes/on turns on verbose tracing to stderr. Replaces the old --debug flag. |
MRG_IOT_TRACE_FILE |
Absolute path collecting that trace in a file instead of stderr, so the terminal stays readable. Ignored unless MRG_IOT_TRACE is on. |
MRG_IOT_TRACE=1 raises the log level to DEBUG and echoes every record to
stderr, so a trace never mixes into piped output; any other value (or unset) is
off. Set MRG_IOT_TRACE_FILE=/abs/path.log to collect that trace in a file
instead, leaving the terminal clean:
MRG_IOT_TRACE=1 MRG_IOT_TRACE_FILE=/tmp/mrg.log mrg-iot exp status myexp
``` Logs are always written to `~/.mrg-iot/debug.log` (appended across runs,
rotated at 5 MB, 3 kept). Attach that file and `mrg-iot --version` to any bug
report.
### Endpoints & known projects
- **gRPC portal:** `grpc.sphere-testbed.net:443` (via `mergetbapi==1.3.41`)
- **SSH jump host:** `jump.sphere-testbed.net:2022`
- **Tunneled ports:** `8554` (RTSP), `9001` (SFTPGo file client — uploads/downloads), `17000` (`ExperimentControl`). Every local port is OS-assigned.
- **Known projects:** `neuiot`, `iotbeta`, `iotdev` (the hardcoded `PROJECTS` list in `mrg_iot/constants.py`; a project outside it is not listed)
---
## Architecture
Three tiers, with a dependency rule enforced by
`tests/test_import_layering.py`:
- **`core/`** — what more than one command needs: reaching an experiment (SSH,
tunnels, session), naming which one to reach, and preparing a command line
before sending it. Never imports a command group.
- **`commands/<group>/`** — one package per subcommand family. Each owns its own
argparse subtree and handlers, and never imports another group.
- **`views/`** — the only place that writes to the terminal. Nothing outside it
calls `print()` or touches `rich` directly.
mrg_iot/ ├── cli.py # a registry: GROUPS + main() dispatcher ├── constants.py # pure data: paths, endpoints, model template, help text ├── core/ │ ├── controller.py # @require_login │ ├── enclave.py # interactive pickers + readiness checks │ ├── ssh.py # XDC SSH connect via the jump host │ ├── tunnel.py # Paramiko port forwarding (ephemeral local ports) │ ├── command_check.py # legacy-syntax rewrite; argument checking (disabled) │ ├── auth.py # credential precedence + login/logout │ ├── api/ # portal gRPC client + retry wrapper │ └── session/ # model, scope, channel, transport, checking, intro ├── commands/ │ ├── cmd/ # run | async-run | wait | status | tasks | shell | ctl │ │ # (+ jobs.py: queued replies; the shared REPL loop) │ ├── devices/ # list | info (both off the GetIntro banner) │ ├── exp/ # exp | realization | materialization | xdc │ ├── file/ # download | upload | list | remove (SFTPGo web client) │ ├── session/ # login | logout │ ├── stream/ # open | list | url (+ the VLC viewer handling) │ └── run.py # the one-command lifecycle ├── models/ # dataclasses + on-disk state ├── views/ # all user-facing output └── utils/ # duration parsing, file helpers
**The `run` workflow** (`commands/run.py`, composing `commands/exp/provision.py`):
Login → Select Project → Select Devices → Create Experiment → Build+Compile+Push Model → Realize → Materialize → Create/Attach XDC → SSH + tunnels + ExperimentControl channel → command loop → Cleanup
Progress is recorded in an `ExpState` dataclass as each resource is created, so
cleanup tears down exactly what exists even when provisioning fails partway.
**Sessions.** `core.session.session_scope` is the single way to open one, for
every command. It is a context manager rather than an open/close pair
specifically so a Ctrl-C partway through building the tunnels cannot leak the
SSH client. Each local tunnel port is assigned by the OS, and URLs pointing at
forwarded services (RTSP, the file server) are built from the live session's
port map — there is no fixed port to hardcode.
---
## Troubleshooting
**SSL / TLS error talking to the portal (common on macOS):**
```bash
export SSL_CERT_FILE="$(python -m certifi)"
export REQUESTS_CA_BUNDLE="$SSL_CERT_FILE"
stream open does nothing / errors: Make sure the vlc binary is installed
and on your PATH (vlc --version). pip/pipx cannot install VLC. If you
would rather use another player, mrg-iot stream url prints the RTSP addresses
and keeps forwarding them.
A file command can't find your file: Names are the file server's, not your
shell's — no globbing, and a path is relative to its root, never to your
machine. Run mrg-iot file list (optionally with a subdirectory) and use the
name it prints.
VLC connects but shows nothing: Check for an editor forwarding the same
port. VS Code's automatic port forwarding binds 127.0.0.1:<port> and shadows
the tunnel for anything connecting to localhost — remove the entry from its
Ports panel, or set "remote.autoForwardPorts": false.
A command hangs or times out: Re-run with MRG_IOT_TRACE=1 and watch the
trace on stderr, or inspect ~/.mrg-iot/debug.log, for the underlying gRPC/SSH
error.
Resources left behind after a crash: Run mrg-iot exp teardown <name> — see
the cleanup sequence. It reports what it removed and
skips whatever is already gone, so it is safe to re-run.
Development
git clone https://gitlab.com/sphere-neu/mrg-iot.git
cd mrg-iot
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
Tests, type-checking, linting, formatting:
pytest tests/ -v # asyncio_mode=auto; no extra flags
pytest tests/test_commands_cmd.py -v # a single file
mypy mrg_iot
pylint mrg_iot
black mrg_iot tests && isort mrg_iot tests
The
devextras (black, pylint, isort, etc.) require Python 3.10+. On a 3.9 environment, install runtime deps only (pip install -e .).
Pinned dependency lockfiles live under requirements/ (requirements/prod.txt
for runtime deps, requirements/dev.txt for the lint/format/type-check tools)
if you'd rather install from those than from pyproject.toml's extras:
pip install -r requirements/prod.txt # runtime only, Python 3.9+
pip install -r requirements/dev.txt # runtime + dev tools, Python 3.10+
Release process
CI (.gitlab-ci.yml) runs pytest on every push, builds an sdist + wheel on
success, and publishes to PyPI on vX.Y.Z tags via OIDC trusted publishing.
The git tag must equal v$(cat mrg_version) — the publish job aborts
otherwise. To cut a release:
# bump the version
echo "1.3.2" > mrg_version
git commit -am "increased version to 1.3.2"
git tag v1.3.2
git push && git push --tags
License
MIT © Sphere / NEU IoT Testbed contributors
Metadata
Release files for mrg-iot 1.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mrg_iot-1.5.0.tar.gz | 281.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mrg_iot-1.5.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 474.3 kB
Release files / mrg_iot-1.5.0.tar.gz
| Download URL | mrg_iot-1.5.0.tar.gz |
|---|---|
| Size | 281.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2dc0524d8a76cc83dc4cd194a8f09fe8afecb957eb18e030f4b697dbecc22552
|
|
BLAKE2b-256 checksum How to use checksums |
c9d47153024610f650f8efc8d3defda8b3f8f56167a4a3d9354e44216e224f82
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Release files / mrg_iot-1.5.0-py3-none-any.whl
| Download URL | mrg_iot-1.5.0-py3-none-any.whl |
|---|---|
| Size | 193.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6a1f469593b27966fb0cbd492181bf9175937448c233606a6df661c74ccd700a
|
|
BLAKE2b-256 checksum How to use checksums |
25ab2638531df36ccf726b79ebb0286f487401d44ea932e38a4d0106db2cb42e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|