Installing Jetstan Agent as a systemd service

This turns the jetstan-agent PyPI package into a real, self-healing Linux daemon: it starts on boot, restarts itself if it crashes or hangs, and pulls + installs new versions automatically when told to over MQTT.

First-time install on a fresh Pi

Two commands. Raspberry Pi OS (Debian 12+) blocks plain pip install system-wide by default (PEP 668) — pipx is the correct tool here, not a workaround: it gives the agent's CLI its own isolated environment, and (unlike a plain pip install --user as root) --global installs put it somewhere every user's $PATH — including sudo's — actually finds it, with no manual symlinking:

sudo apt update && sudo apt install -y pipx
sudo pipx install --global jetstan-agent

Then run the installer, supplying your MQTT credentials directly so the service is fully configured and running after this one command — no SSH round-trip to edit a file:

sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password '<password>'

(Single-quote the password — protects special characters like ! from the shell.)

--mqtt-password is visible in shell history and briefly in ps output while the command runs. If that matters for your setup (shared machine, scripted provisioning, etc.), pipe it in instead with --mqtt-password-stdin:

echo '<password>' | sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password-stdin

Either way, the password ends up in /etc/jetstan/agent.env (mode 600, owned by the jetstan service account) — that part is unavoidable, the running service has to read it from somewhere.

That's it. The installer creates a dedicated jetstan system user, a venv at /opt/jetstan/venv, /etc/jetstan/agent.env (populated with what you passed), and the jetstan-agent systemd service — enabled, started, and verified actually connected before the command returns 0.

jetstan-agent status
jetstan-agent logs

Reboot the Pi whenever you like afterward — the service comes back on its own, no manual commands required.

Prefer to configure credentials by hand instead?

Omit all three --mqtt-* flags:

sudo jetstan-agent-install
sudo nano /etc/jetstan/agent.env   # fill in MQTT_HOST / MQTT_USERNAME / MQTT_PASSWORD
sudo systemctl restart jetstan-agent

On a brand-new install with no credentials yet, the service failing to connect until you fill them in is expected, not an error.

Troubleshooting

• sudo: jetstan-agent-install: command not found after pipx install --global — you're on a pipx older than 1.4 (check pipx --version; sudo apt install -y pipx on current Raspberry Pi OS is well past this). Upgrade pipx, or fall back to sudo pipx install jetstan-agent and manually symlink the three scripts pipx prints the path to (under /root/.local/bin/) into /usr/local/bin/.
• A fresh pip3 install jetstan-agent fails with an externally-managed-environment error — expected on current Debian; this is exactly why the instructions above use pipx, not pip3, directly.

Everyday operations

jetstan-agent status     # systemctl status, under the hood
jetstan-agent start
jetstan-agent stop
jetstan-agent restart
jetstan-agent logs        # journalctl -u jetstan-agent -f
jetstan-agent logs -n 200 --no-follow

(jetstan-agent run also exists — that's the foreground agent process itself, which is what the systemd unit's ExecStart= invokes. You normally never call it directly.)

How the OTA update loop works

There is no separate download/artifact server — the existing PyPI distribution channel is the update mechanism. The agent is purely event-driven: it never polls PyPI on its own, it only acts when it receives a specific MQTT message.

On your dev machine, jetstan-cli's jetstan_deploy.py does the whole publish side in one command:

cd jetstan-cli && pip install -e ".[dev,deploy]"    # once, installs build + twine
python -m jetstan_cli.jetstan_deploy

It: reads the version from the root pyproject.toml → refuses to proceed if that version is already published → runs the test suite → rebuilds dist/ from a clean slate → uploads to PyPI via twine (your credentials, your prompt, same as running twine by hand) → waits (default 45s, see --wait-seconds) for PyPI's index to actually propagate → publishes {"type": "ota.available", "version": "..."} to your configured MQTT topic, using the same MQTTConfig.from_env() the agent itself uses (so it reads from your local .env — see the root README.md).

Run python -m jetstan_cli.jetstan_deploy --dry-run to build and test without publishing anything, or --yes to skip the confirmation prompt. This performs a real, public PyPI upload and pushes an update to every Pi listening on the topic — run it yourself, deliberately, not as part of an unattended script.

On each Pi, once that message arrives:

1. The running agent receives it, runs pip install --upgrade jetstan-agent==1.4.0 inside its own venv, and then deliberately exits.
2. systemd's Restart=always immediately relaunches the service — now running the newly installed version.

If the pip upgrade fails (bad version, network blip), the agent logs the error and keeps running the current version — nothing is torn down. If the message names a version that's already running, it's a no-op (safe against duplicate/retained MQTT deliveries). If the upgrade succeeds but the new version turns out to be broken, see "A bad release can't take a device permanently offline" below — that's handled too.

Uninstalling

sudo jetstan-agent-uninstall                          # stops/disables the service, keeps config & venv
sudo jetstan-agent-uninstall --purge-config            # also deletes /etc/jetstan (credentials)
sudo jetstan-agent-uninstall --purge-state             # also deletes /opt/jetstan and the jetstan user
sudo jetstan-agent-uninstall --purge-config --purge-state   # full removal

Reliability details

• Restart=always + RestartSec=5: any crash or clean exit restarts the service after 5s.
• WatchdogSec=60: the agent pings systemd every ~30s while its MQTT loop is alive; if it ever hangs (not just crashes), systemd restarts it too.
• StartLimitBurst=8 / StartLimitIntervalSec=120: if something is genuinely broken and it crash-loops more than 8 times in 2 minutes, systemd stops trying rather than looping forever. Set deliberately above the automatic-rollback threshold below, with margin — see there for why the exact numbers matter.
• The service runs as an unprivileged jetstan user with heavy systemd sandboxing (ProtectSystem=strict, no new privileges, no device/kernel access, etc.) — the only writable path is its own venv, which is what lets self-upgrade work without weakening anything else.

A bad release can't take a device permanently offline

Restart=always on its own has a gap: if a newly installed release crashes on startup, systemd just keeps relaunching the same broken code, burns through its restart budget in under a minute, then gives up entirely — the unit sits dead, unreachable by design (SSH is your only way back in at that point). That defeats the actual point of having an OTA channel.

ExecStart= doesn't invoke the agent directly — it invokes /opt/jetstan/supervisor.py, a small file jetstan-agent-install writes once, deliberately outside /opt/jetstan/venv, the only path pip install --upgrade jetstan-agent ever touches. That's what lets it survive a release that breaks at import time, which a fix living inside the upgraded package itself couldn't. It launches the real agent as a child process; if the same version fails 3 times in a row, it automatically pip installs back to the previously-running version instead of retrying the broken one a 4th time, and restarts into it. A version is only considered "safe" once it successfully connects to MQTT — a version that connects fine and crashes hours later for an unrelated reason is never rolled back; only a version that never proved itself healthy in the first place is a rollback candidate. If the rollback target also keeps failing, the supervisor stops intervening and lets it settle into systemd's normal failed state — an honest, loudly-logged signal for a human to SSH in, not an infinite ping-pong between two bad versions or a silent brick either way.

Every OTA push only ever touches this package's own code

pip install --upgrade jetstan-agent runs with --no-deps, and every dependency in pyproject.toml is pinned to an exact version, not a range. Without both of these, a routine push could silently pull in a new transitive dependency release (e.g. paho-mqtt) that was never tested against this codebase — on a push that didn't even touch jetstan-agent's own code. Bumping a dependency is now a deliberate, tested part of a normal release (bump the pin, jetstan-cli's jetstan_deploy.py test gate covers it) rather than something that can change underneath a live fleet on its own schedule.

Every device gets its own MQTT identity by default

Two devices sharing one MQTT client_id isn't a hypothetical — it's already happened once during this project's own testing: the broker disconnects whichever connection is older the instant a second one claims the same ID. The default client_id is now derived per-machine (from /etc/machine-id), not a fixed string, so this can't happen from two otherwise-identical installs. Set JETSTAN_MQTT_CLIENT_ID explicitly in agent.env only if you need a specific, human-meaningful ID (e.g. matching a fleet inventory system) — the default is safe to leave alone.

Arduino firmware updates (optional)

The installer always adds the jetstan service account to the dialout group and the systemd unit always leaves real /dev visible (PrivateDevices is not set) — both needed for USB-serial access, and both dormant/unused unless you set ARDUINO_ENABLED=true in agent.env. See README.md for the full variable list and how the automatic compile-and-flash pipeline works.

Zero-touch: jetstan-agent-install provisions the whole toolchain

avrdude and arduino-cli are provisioned automatically, by the installer itself, as part of the same one sudo jetstan-agent-install run described at the top of this file — no separate manual step, no curl | sh. Specifically (see jetstan_agent/arduino_provision.py):

• avrdude is installed via apt-get install avrdude (root, one-time).
• arduino-cli is installed by downloading a pinned release archive directly from its GitHub Releases page, together with that release's own published _checksums.txt, verifying the archive's SHA-256 against it, and only then extracting the single binary to /usr/local/bin/arduino-cli — never a piped-to-shell installer script. A checksum mismatch is a hard failure, not a warning.

Both steps are best-effort during install: a failure (typically, no network reachable at install time) is logged clearly but does not fail the overall jetstan-agent-install run — the always-needed OTA/MQTT agent still installs and starts. If you enable ARDUINO_ENABLED=true later without these having succeeded, the runtime's own validate_environment step (see software/deployment/bootstrap.py) is the real, hard gate — it fails loudly, with a specific reason, and blocks firmware compile/flash/telemetry until it's resolved.

To retry provisioning without a full reinstall:

sudo python3 -c "from jetstan_agent import arduino_provision as p; p.ensure_avrdude_installed(); p.ensure_arduino_cli_installed()"

(using the venv's own interpreter, e.g. /opt/jetstan/venv/bin/python3, so the installed jetstan_agent package is importable.)

AVR core install location — the jetstan service account has no home directory (useradd --no-create-home), and arduino-cli (a Go binary) resolves several of its own paths (config file, build cache) straight from $HOME/XDG variables, which has nowhere valid to go for a homeless account. jetstan-agent-install already creates /var/lib/jetstan (via systemd's StateDirectory=jetstan); the runtime's own bootstrap_environment step points arduino-cli at /var/lib/jetstan/arduino-cli/ for all of this automatically — no manual mkdir/chown/config-file authoring needed (this replaces what used to be a manual step here).

You do not need to run arduino-cli core install arduino:avr yourself. Once arduino-cli and avrdude are present (installed automatically above, or manually — see the fallback below), the deployment DAG's bootstrap_environment task (see software/deployment/bootstrap.py and arduino_updates/provisioner.py) installs and verifies the arduino:avr core itself, automatically, on every startup — as the unprivileged jetstan user, no root involved. It's idempotent: already-correct is a fast no-op, and it never reinstalls on every OTA push, only when the core is genuinely missing or (if ARDUINO_CORE_VERSION is set in agent.env) at the wrong version. The network-dependent steps (fetching the index, downloading the core) are retried a bounded number of times with backoff — a transient outage recovers on its own; a host with no network at all fails the bootstrap_environment DAG task loudly rather than hanging.

What stays manual by design, even with automatic provisioning: the decision to trust and run apt/a downloaded binary at all happens only during the root-privileged, human-invoked jetstan-agent-install run — never from the OTA-triggered, unprivileged jetstan process, which has no path to root and never attempts either install itself. This is the same security boundary as before; only where the automation lives changed (installer, not "a step INSTALL.md tells you to run by hand").

Manual fallback

If automatic provisioning fails (no network at install time, an unsupported architecture, or you simply prefer to control it yourself):

sudo apt install avrdude
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sudo BINDIR=/usr/local/bin sh

Everything downstream of that (AVR core provisioning, compiling, flashing) still happens automatically exactly as described above — this fallback only replaces the two binaries the installer otherwise fetches for you.

Known limitation

The installer uses useradd/groupadd/userdel (standard on Debian-based systems, including Raspberry Pi OS). It hasn't been adapted for Yocto/busybox -based images, which use different user-management tools.