krabby-bench

Watches ECR for new locomotion images, updates the stack when one appears, runs a firmware smoke test, and alerts on failure.

Install

Prerequisite: krabby-launcher must be installed and krabby install run before krabby-bench — the watchdog calls the krabby CLI to pull images and flash firmware. Follow top-level README §1–2 (user venv + sudo -E env PATH="$PATH" "$(which krabby)" install), then add the bench package:

source ~/.venv-krabby/bin/activate
pip install krabby-bench

Alternate (system-wide, matches scripts/jetson/bootstrap.sh): sudo pip3 install krabby-launcher krabby-bench then sudo krabby install.

Then run the install command as root to configure and start the watchdog service:

sudo krabby-bench install [options]

Options

Flag	Default	Description
--ssm-prefix	/krabby/bench	SSM parameter path prefix. When set, credentials are loaded from AWS SSM.
--ecr-tag	main-latest	ECR image tag to watch.
--firmware-channel	release/0.2.9	Firmware channel for smoke tests.
--error-alert-type	both	Alert delivery type: email, github, or both.
--github-repo	(none)	owner/repo to open issues against (legacy mode only).

SSM mode (recommended for fleet use)

Pass IAM access keys at install time; the service fetches credentials from SSM at runtime and re-reads them automatically on each refresh interval.

sudo \
  BENCH_AWS_KEY_ID=AKIA... \
  BENCH_AWS_SECRET_KEY=... \
  krabby-bench install \
    [--ssm-prefix /krabby/bench] \
    [--ecr-tag main-latest] \
    [--firmware-channel release/0.2.9] \
    [--error-alert-type both]

install writes /etc/krabby-bench/config.toml, then enables and starts the service.

SSM parameter layout

Create these in AWS SSM Parameter Store before or after installing. The service starts without them and logs a single warning; it picks them up automatically within one credentials_refresh_interval (default: 3600 s) once they exist.

Path	Type	Description
/krabby/bench/smtp-host	String	SMTP server hostname
/krabby/bench/smtp-port	String	SMTP port (default 587)
/krabby/bench/smtp-user	String	SMTP login username
/krabby/bench/smtp-password	SecureString	SMTP login password
/krabby/bench/smtp-from	String	From address
/krabby/bench/smtp-to	String	Alert recipient address
/krabby/bench/github-repo	String	owner/repo to open issues against
/krabby/bench/github-token	SecureString	Fine-grained PAT with Issues write scope

IAM policy

The IAM user whose access key is passed to install needs only:

{
  "Effect": "Allow",
  "Action": "ssm:GetParametersByPath",
  "Resource": "arn:aws:ssm:*:*:parameter/krabby/bench/*"
}

Credential rotation

Update values in SSM. Devices pick up the new credentials within one poll interval — no SSH required.

To rotate the AWS access key, re-run install with the new key:

sudo BENCH_AWS_KEY_ID=AKIANEW... BENCH_AWS_SECRET_KEY=... \
  krabby-bench install --ssm-prefix /krabby/bench

Legacy mode

Pass credentials via environment variables. Written to /etc/krabby-bench/smtp.env (mode 600) and loaded by the systemd unit.

sudo \
  BENCH_SMTP_HOST=smtp.example.com \
  BENCH_SMTP_PORT=587 \
  BENCH_SMTP_USER=krabby-errors@example.com \
  BENCH_SMTP_PASSWORD=secret \
  BENCH_SMTP_FROM=krabby-errors@example.com \
  BENCH_SMTP_TO=krabby-errors@example.com \
  BENCH_GITHUB_REPO=owner/krabby-research \
  BENCH_GITHUB_TOKEN=ghp_... \
  krabby-bench install [--ecr-tag main-latest] [--firmware-channel release/0.2.9] [--error-alert-type both]

Legacy environment variables

Variable	Required for	Description
BENCH_SMTP_HOST	email alerts	SMTP server hostname
BENCH_SMTP_PORT	email alerts	SMTP port (default 587)
BENCH_SMTP_USER	email alerts	SMTP login username
BENCH_SMTP_PASSWORD	email alerts	SMTP login password
BENCH_SMTP_FROM	email alerts	From address
BENCH_SMTP_TO	email alerts	Alert recipient address
BENCH_GITHUB_REPO	GitHub alerts	owner/repo to open issues against
BENCH_GITHUB_TOKEN	GitHub alerts	Fine-grained PAT with Issues write scope

Config

Non-secret fields only — credentials come from SSM or the env vars above.

Default path: /etc/krabby-bench/config.toml

[ecr]
repo = "public.ecr.aws/t7t7b3i3/krabby-locomotion"
tag = "main-latest"
poll_interval = 60          # seconds

[smoke]
firmware_channel = "release/0.2.9"
run_hal_check = false

[alert]
mode = "both"               # "email" | "github" | "both"
dedup_window = 3600         # suppress repeat alerts for the same failure (seconds)

[github]
repo = "owner/krabby-research"

[ssm]
prefix = "/krabby/bench"
credentials_refresh_interval = 3600   # how often to re-fetch from SSM (seconds)

Smoke test

For each new digest the watchdog:

1. Runs krabby firmware show to discover attached board ports.
2. Runs krabby firmware update <channel> <port> for each port.
3. Runs krabby firmware show again and parses the version strings.
4. Asserts all three boards report the same version.
5. Fetches https://krabby-firmware-public.s3.amazonaws.com/<channel>/latest.json and checks the version matches the S3 manifest.

Monitor

journalctl -fu krabby-bench

For verbose output on a one-off manual run:

krabby-bench --log-level DEBUG

When running as a systemd service, the log level must be set in the unit file — systemd launches the process with a fixed command line, so flags can't be passed after the fact. Edit /etc/systemd/system/krabby-bench.service:

ExecStart=/usr/local/bin/krabby-bench --log-level DEBUG --config /etc/krabby-bench/config.toml

Then apply and restart:

sudo systemctl daemon-reload && sudo systemctl restart krabby-bench

Valid levels: DEBUG, INFO (default), WARNING, ERROR.

Force a recheck

To make the watchdog re-run the update + smoke test immediately — regardless of whether the ECR digest has changed — run:

sudo krabby-bench force-recheck

This signals the running service to clear its last-tested digest. The recheck runs on the next poll cycle (within one poll_interval).

Force a failure (test alert path)

Unplug one Mega. Clear the state file to trigger a re-test on the next poll:

sudo bash -c 'echo "{}" > /var/lib/krabby-bench/state.json'
sudo systemctl restart krabby-bench

Within one poll cycle the watchdog detects the failure and fires an alert.

State file

/var/lib/krabby-bench/state.json — persists the last-tested digest and last-alert metadata. Clear it to force a re-test on the next poll.

Four-stage harness (install → flash → bringup → motion)

One-shot CI path for cold-start verification. Extends this package rather than a second bench service. Dual-use Orin: uses ~/.venv-krabby-bench and scripts/jetson/bench-reset.sh (never deletes ~/.venv-krabby).

# From cold-start or any venv that has this package (editable ok):
cd /path/to/krabby-research
source ~/.venv-krabby/bin/activate   # or bench venv after first install
pip install -e ./bench

# Full run (reset + fresh Install + Flash + Bringup + Motion).
# Requires sudo for krabby install; keep one leg free to move.
krabby-bench harness --repo-root "$(pwd)"

# After Install already proved — skip wipe/reinstall:
krabby-bench harness --skip-install --no-reset --repo-root "$(pwd)"

Stage	What it does
install	bench-reset.sh → create ~/.venv-krabby-bench → pip install krabby-launcher (+ local ./bench) → sudo -E env PATH=<venv/bin>:$PATH <venv/bin/krabby> install --no-launch-on-startup
flash	Existing run_smoke: flash all boards, assert versions match S3
bringup	krabby run --gamepad-only; wait for container + MCU connected in logs
motion	Stop container; jog RLKL, RRKL, FLKL, FRHL, FRKL (override with --joint …) via host repo firmware/ (PIN_REV=2 is local-testing only / dev-local — not S3; will not be used going forward); assert pot or hall delta per joint

Motion stops the stack temporarily because Jetson HAL observations do not carry real pot/hall (KNOWN-ISSUES #6/#7). Failure output names the stage and the command that failed.

Useful flags: --skip-flash, --skip-bringup, --skip-motion, --firmware-channel, --joint, --rmi (pass through to reset), --no-discord, --run-url, --commit.

Discord: set DISCORD_WEBHOOK_URL to post PASS/FAIL after the harness (skipped if unset). In GitHub Actions the same secret name is used.

Flash note (PIN_REV=2 / local only)

PIN_REV=2 is for local Uno v0.1 testing only and will not be used going forward. Those boards report dev-local, so the harness does not take the published S3 firmware (wrong pin map) and skips S3 overwrite. Future CI should use the published pin revision / S3 artifact path.

CI: artifact health (hosted runners)

Workflow: .github/workflows/artifact-health.yml

Trigger (not on push): GitHub starts this workflow only after Build and push locomotion image to ECR completes successfully (workflow_run). Manual Actions → Artifact health → Run workflow still works.

Why wait: Artifact health does not build the locomotion image — it pulls the new one from ECR (main-latest / release-latest) and smokes it. Running in parallel with publish would race (pull a missing or stale tag). Waiting until publish succeeds means the pull is of the image that push just produced. Packages and firmware S3 checks run in the same workflow afterward (they wait on publish too; that is acceptable). If locomotion publish fails, Artifact health does not run.

Job	Check
packages	pip install + import for krabby-launcher, krabby-firmware, krabby-bench
locomotion-image	pull main-latest and release-latest, docker run … --help must show --teleop-ip (arm64 via QEMU)
firmware-artifact	S3 latest.json + manifest ver_string + HEX HEAD for release/0.2.15
notify	Discord summary (skipped if DISCORD_WEBHOOK_URL secret unset). FAIL always; consecutive PASSes collapsed (PASS only after a non-green prior run)

Publish firmware / locomotion also Discord on failure (same secret) so a red publish is not silent when Artifact health never starts. Messages include failed step names when resolvable.

If a run is red (or Discord reports FAIL): docs/CI-FAILURES.md.

CI: self-hosted Orin runner

Full procedure (every bench Orin): SELF-HOSTED-RUNNER.md

Workflow: .github/workflows/bench-harness.yml

The hardware harness job is gated on repo variable BENCH_RUNNER_ENABLED=true so it does not queue forever before a runner exists. Until then a stub job explains the skip.

Summary: install linux-arm64 Actions runner under ~/actions-runner (not inside the git tree) with labels self-hosted,krabby-bench; registration token needs repo admin and expires ~1h; then set BENCH_RUNNER_ENABLED=true and optional DISCORD_WEBHOOK_URL. See the doc above for checksums, svc.sh, and failure modes.

Until the runner is registered, exercise hardware locally:

krabby-bench harness --skip-install --no-reset --repo-root "$(pwd)"
# optional: export DISCORD_WEBHOOK_URL=… to prove notifications

Local development

Set BENCH_SMTP_* and BENCH_GITHUB_TOKEN env vars directly; the watchdog reads them as fallback when no SSM prefix is configured:

BENCH_SMTP_HOST=smtp.example.com ... python -m krabby_bench.watchdog