Installing Jetstan Agent as a systemd service

This turns the jetstan-agent PyPI package into a real, self-healing Linux daemon: it starts on boot, restarts itself if it crashes or hangs, and pulls + installs new versions automatically when told to over MQTT.

First-time install on a fresh Pi

Two commands. Raspberry Pi OS (Debian 12+) blocks plain pip install system-wide by default (PEP 668) — pipx is the correct tool here, not a workaround: it gives the agent's CLI its own isolated environment, and (unlike a plain pip install --user as root) --global installs put it somewhere every user's $PATH — including sudo's — actually finds it, with no manual symlinking:

sudo apt update && sudo apt install -y pipx
sudo pipx install --global jetstan-agent

Then run the installer, supplying your MQTT credentials directly so the service is fully configured and running after this one command — no SSH round-trip to edit a file:

sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password '<password>'

(Single-quote the password — protects special characters like ! from the shell.)

--mqtt-password is visible in shell history and briefly in ps output while the command runs. If that matters for your setup (shared machine, scripted provisioning, etc.), pipe it in instead with --mqtt-password-stdin:

echo '<password>' | sudo jetstan-agent-install \
  --mqtt-host <your-cluster>.hivemq.cloud \
  --mqtt-username <username> \
  --mqtt-password-stdin

Either way, the password ends up in /etc/jetstan/agent.env (mode 600, owned by the jetstan service account) — that part is unavoidable, the running service has to read it from somewhere.

That's it. The installer creates a dedicated jetstan system user, a venv at /opt/jetstan/venv, /etc/jetstan/agent.env (populated with what you passed), and the jetstan-agent systemd service — enabled, started, and verified actually connected before the command returns 0.

jetstan-agent status
jetstan-agent logs

Reboot the Pi whenever you like afterward — the service comes back on its own, no manual commands required.

Prefer to configure credentials by hand instead?

Omit all three --mqtt-* flags:

sudo jetstan-agent-install
sudo nano /etc/jetstan/agent.env   # fill in MQTT_HOST / MQTT_USERNAME / MQTT_PASSWORD
sudo systemctl restart jetstan-agent

On a brand-new install with no credentials yet, the service failing to connect until you fill them in is expected, not an error.

Troubleshooting

• sudo: jetstan-agent-install: command not found after pipx install --global — you're on a pipx older than 1.4 (check pipx --version; sudo apt install -y pipx on current Raspberry Pi OS is well past this). Upgrade pipx, or fall back to sudo pipx install jetstan-agent and manually symlink the three scripts pipx prints the path to (under /root/.local/bin/) into /usr/local/bin/.
• A fresh pip3 install jetstan-agent fails with an externally-managed-environment error — expected on current Debian; this is exactly why the instructions above use pipx, not pip3, directly.

Everyday operations

jetstan-agent status     # systemctl status, under the hood
jetstan-agent start
jetstan-agent stop
jetstan-agent restart
jetstan-agent logs        # journalctl -u jetstan-agent -f
jetstan-agent logs -n 200 --no-follow

(jetstan-agent run also exists — that's the foreground agent process itself, which is what the systemd unit's ExecStart= invokes. You normally never call it directly.)

How the OTA update loop works

There is no separate download/artifact server — the existing PyPI distribution channel is the update mechanism. The agent is purely event-driven: it never polls PyPI on its own, it only acts when it receives a specific MQTT message.

On your dev machine, jetstan-cli's jetstan_deploy.py does the whole publish side in one command:

cd jetstan-cli && pip install -e ".[dev,deploy]"    # once, installs build + twine
python -m jetstan_cli.jetstan_deploy

It: reads the version from the root pyproject.toml → refuses to proceed if that version is already published → runs the test suite → rebuilds dist/ from a clean slate → uploads to PyPI via twine (your credentials, your prompt, same as running twine by hand) → waits (default 45s, see --wait-seconds) for PyPI's index to actually propagate → publishes {"type": "ota.available", "version": "..."} to your configured MQTT topic, using the same MQTTConfig.from_env() the agent itself uses (so it reads from your local .env — see the root README.md).

Run python -m jetstan_cli.jetstan_deploy --dry-run to build and test without publishing anything, or --yes to skip the confirmation prompt. This performs a real, public PyPI upload and pushes an update to every Pi listening on the topic — run it yourself, deliberately, not as part of an unattended script.

On each Pi, once that message arrives:

1. The running agent receives it, runs pip install --upgrade jetstan-agent==1.4.0 inside its own venv, and then deliberately exits.
2. systemd's Restart=always immediately relaunches the service — now running the newly installed version.

If the pip upgrade fails (bad version, network blip), the agent logs the error and keeps running the current version — nothing is torn down. If the message names a version that's already running, it's a no-op (safe against duplicate/retained MQTT deliveries). If the upgrade succeeds but the new version turns out to be broken, see "A bad release can't take a device permanently offline" below — that's handled too.

Uninstalling

sudo jetstan-agent-uninstall                          # stops/disables the service, keeps config & venv
sudo jetstan-agent-uninstall --purge-config            # also deletes /etc/jetstan (credentials)
sudo jetstan-agent-uninstall --purge-state             # also deletes /opt/jetstan and the jetstan user
sudo jetstan-agent-uninstall --purge-config --purge-state   # full removal

Reliability details

• Restart=always + RestartSec=5: any crash or clean exit restarts the service after 5s.
• WatchdogSec=60: the agent pings systemd every ~30s while its MQTT loop is alive; if it ever hangs (not just crashes), systemd restarts it too.
• StartLimitBurst=8 / StartLimitIntervalSec=120: if something is genuinely broken and it crash-loops more than 8 times in 2 minutes, systemd stops trying rather than looping forever. Set deliberately above the automatic-rollback threshold below, with margin — see there for why the exact numbers matter.
• The service runs as an unprivileged jetstan user with heavy systemd sandboxing (ProtectSystem=strict, no new privileges, no device/kernel access, etc.) — the only writable path is its own venv, which is what lets self-upgrade work without weakening anything else.

A bad release can't take a device permanently offline

Restart=always on its own has a gap: if a newly installed release crashes on startup, systemd just keeps relaunching the same broken code, burns through its restart budget in under a minute, then gives up entirely — the unit sits dead, unreachable by design (SSH is your only way back in at that point). That defeats the actual point of having an OTA channel.

ExecStart= doesn't invoke the agent directly — it invokes /opt/jetstan/supervisor.py, a small file jetstan-agent-install writes once, deliberately outside /opt/jetstan/venv, the only path pip install --upgrade jetstan-agent ever touches. That's what lets it survive a release that breaks at import time, which a fix living inside the upgraded package itself couldn't. It launches the real agent as a child process; if the same version fails 3 times in a row, it automatically pip installs back to the previously-running version instead of retrying the broken one a 4th time, and restarts into it. A version is only considered "safe" once it successfully connects to MQTT — a version that connects fine and crashes hours later for an unrelated reason is never rolled back; only a version that never proved itself healthy in the first place is a rollback candidate. If the rollback target also keeps failing, the supervisor stops intervening and lets it settle into systemd's normal failed state — an honest, loudly-logged signal for a human to SSH in, not an infinite ping-pong between two bad versions or a silent brick either way.

Every OTA push only ever touches this package's own code

pip install --upgrade jetstan-agent runs with --no-deps, and every dependency in pyproject.toml is pinned to an exact version, not a range. Without both of these, a routine push could silently pull in a new transitive dependency release (e.g. paho-mqtt) that was never tested against this codebase — on a push that didn't even touch jetstan-agent's own code. Bumping a dependency is now a deliberate, tested part of a normal release (bump the pin, jetstan-cli's jetstan_deploy.py test gate covers it) rather than something that can change underneath a live fleet on its own schedule.

Every device gets its own MQTT identity by default

Two devices sharing one MQTT client_id isn't a hypothetical — it's already happened once during this project's own testing: the broker disconnects whichever connection is older the instant a second one claims the same ID. The default client_id is now derived per-machine (from /etc/machine-id), not a fixed string, so this can't happen from two otherwise-identical installs. Set JETSTAN_MQTT_CLIENT_ID explicitly in agent.env only if you need a specific, human-meaningful ID (e.g. matching a fleet inventory system) — the default is safe to leave alone.

Arduino firmware updates (optional)

The installer always adds the jetstan service account to the dialout group and the systemd unit always leaves real /dev visible (PrivateDevices is not set) — both needed for USB-serial access, and both dormant/unused unless you set ARDUINO_ENABLED=true in agent.env. See README.md for the full variable list and how the automatic compile-and-flash pipeline works.

One-time setup per Pi — install the compiler and flasher. This is deliberately a manual, human-run step (not something the OTA pipeline ever does on its own — see README.md for why):

sudo apt install avrdude
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sudo BINDIR=/usr/local/bin sh

Important — install the AVR core into a location the service can actually read. The jetstan service account has no home directory (useradd --no-create-home), and arduino-cli (a Go binary) resolves several of its own paths — its config file, and separately things like its build cache — straight from $HOME or the XDG base-directory variables, independent of each other. None of that has anywhere valid to resolve to for a homeless account, and fails with permission denied. Give it an explicit, shared location instead — jetstan-agent-install must have already been run at least once so /var/lib/jetstan exists:

sudo mkdir -p /var/lib/jetstan/arduino-cli
sudo tee /var/lib/jetstan/arduino-cli/arduino-cli.yaml > /dev/null <<'EOF'
directories:
  data: /var/lib/jetstan/arduino-cli/data
  downloads: /var/lib/jetstan/arduino-cli/staging
  user: /var/lib/jetstan/arduino-cli/user
EOF
sudo arduino-cli --config-file /var/lib/jetstan/arduino-cli/arduino-cli.yaml core install arduino:avr
sudo chown -R jetstan:jetstan /var/lib/jetstan/arduino-cli

The agent is already configured to pass this same --config-file on every compile, and to run arduino-cli with $HOME/$XDG_CACHE_HOME/ $XDG_CONFIG_HOME all pointed at this same directory (see compiler.py's _build_subprocess_env and ArduinoConfig.arduino_cli_config_file / ARDUINO_CLI_CONFIG_FILE if you ever need to point it elsewhere) — so as long as the core was installed into this exact location and chowned to jetstan as shown above, no further configuration is needed.

After this, every future push that changes the Arduino sketch compiles and flashes automatically — no further manual steps on the Pi.

Known limitation

The installer uses useradd/groupadd/userdel (standard on Debian-based systems, including Raspberry Pi OS). It hasn't been adapted for Yocto/busybox -based images, which use different user-management tools.