Skip to main content

Cyberwave logo

Cyberwave Edge Core

This module is part of Cyberwave: Making the physical world programmable.

Cyberwave Edge Core acts as the orchestrator of Cyberwave edge drivers.

License Documentation Discord PyPI version PyPI Python versions Release to PyPI

Quickstart

SSH to the edge device where you want to install Edge Core, then install the Cyberwave CLI and run the installer:

# Install the Cyberwave CLI (one-time setup)
curl -fsSL https://cyberwave.com/install.sh | bash

# Run the edge installer (interactive)
sudo cyberwave edge install

The installer will prompt you to log in with your Cyberwave account, select a workspace and environment, and persist configuration under ~/.cyberwave/. You can override the config directory via the CYBERWAVE_EDGE_CONFIG_DIR environment variable. Legacy installs that used /etc/cyberwave are automatically migrated.

Permissions on the config directory:

  • credentials.json is written with mode 0600 (owner-only) because it holds your API token.
  • fingerprint.json is written with mode 0644 (world-readable) because the device fingerprint is a hardware identifier, not a secret. This lets user shells read it even when Edge Core runs as root via systemd.
  • On systemd deployments where Edge Core runs as root, the service re-chowns files under ~/.cyberwave/ on startup so they stay owned by the user whose home directory holds them.

Don't have a Cyberwave account? Get one at cyberwave.com

Config files created

The installer and Edge Core create these files in the config directory:

File Description
credentials.json API token and workspace information
fingerprint.json Device fingerprint (generated by Edge Core)
environment.json Selected environment and twin UUIDs
edge.json Cached edge device record from the backend (written on registration, refreshed periodically)
{twin-uuid}.json Cached twin + asset record (one per twin)
cameras.json Deprecated. Legacy camera device config; superseded by edge.json metadata.cameras

Edge Core requires credentials.json to operate. fingerprint.json is produced by Edge Core; environment.json is written by the CLI during setup. edge.json is written on registration and stores the full edge device record, including metadata.cameras (the device list and the twin-to-device mapping). Every ~5 minutes reconcile_edge_json_file_sync fetches the record again, but it rewrites the file only when metadata.cameras changed, because a rewrite is what the camera drift check reacts to (see Camera pins: drift and missing devices).

How Edge Core works

On startup (service or direct run), Edge Core performs the following steps:

  1. Validate credentials from credentials.json.
  2. Connect to the backend MQTT broker and verify connectivity.
  3. Start a bootstrap health publisher that sends periodic edge health messages while drivers are starting up.
  4. Register the edge device and record a unique edge_fingerprint.
  5. Download the selected environment and resolve twins linked to the fingerprint.
  6. Start drivers for linked twins. Special handling for attached camera child twins:
    • If a twin is a camera child (has attach_to_twin_uuid), Edge Core does not start a separate driver for it.
    • Camera child UUIDs are passed to the parent driver via CYBERWAVE_CHILD_TWIN_UUIDS.
  7. Stop the bootstrap health publisher once drivers are running (drivers publish their own health messages; keeping both would produce duplicate signals in the UI).
  8. Pull workflow workers (wf_*.py) for the twins listed in environment.json and start the worker container (if any worker files exist in {config_dir}/workers/).

Scope of workflow worker sync

Workflow worker sync is scoped strictly to the twin UUIDs the operator selected at install time and persisted to environment.json under twin_uuids. This is intentionally narrower than the fingerprint-based discovery used for drivers and the bootstrap health publisher: an environment can carry stale metadata.edge_fingerprint entries from previous installs, and we don't want those to pull unrelated wf_*.py files onto this edge. For backward compatibility, installs that predate the twin_uuids field still fall back to fingerprint-based discovery.

During driver startup, Docker image pull progress is mirrored into the edge-core service logs and forwarded through the same MQTT-backed driver log stream used for runtime container logs, so users can follow image download progress remotely.

Remote restart (Edge REST API)

Request a remote restart of Edge Core via the REST API:

POST /api/v1/edges/{uuid}/restart-core

The API will publish an MQTT message to the edge's command topic:

Topic: edges/{edge_uuid}/command

Example payload:

{ "command": "restart_edge_core" }

When Edge Core receives this command it performs a graceful restart consisting of:

  1. Stopping the worker container (if running).
  2. Removing cached twin JSON files from the edge config directory.
  3. Stopping and removing any edge-managed driver containers, then pruning stopped containers.
  4. Resolving any active driver_starting alerts on the affected twins so leftovers from the previous run do not stay visible after the restart.
  5. Re-downloading the selected environment and restarting drivers.
  6. Restarting the worker container (if worker files exist).

The restart deletes the cached twin JSON files without a final push, so an edit a driver made to its twin file after the last runtime tick (see Twin JSON file) is lost; everything else is downloaded again from the backend.

Each driver startup attempt creates a driver_starting twin alert that tracks the in-flight startup (image pull, container launch, post-launch health probe). The alert is automatically resolved once the driver container is observed running; if the attempt fails, it's annotated with the failure phase (e.g. pull_failed / pull_timeout / pull_oserror), escalated to critical severity, and resolved. The same lifecycle applies to the ML worker via the worker_starting alert, which is raised before the image pull and held until the worker container is observed running — so a worker whose image is already on disk still reports that it is starting, rather than coming up silently. The alert is therefore guaranteed to clear when the driver has restarted; longer-lived failure conditions are surfaced through separate driver_start_failure alerts created by the orchestrator.

While docker pull is running, edge-core writes pull progress directly onto the alert's metadata (progress_percent, downloaded_bytes / total_bytes, downloaded_human / total_human, phase) so the workbench renders e.g. Downloading driver image (cyberwaveos/ugv-driver:dev) — 43% instead of a stale "Still pulling" spinner. The byte counts join the message once every layer has declared its size (745 MB of 1.55 GB); until then the image size is not yet known and only the percentage is shown. The same summary feeds the systemd watchdog (systemctl status cyberwave-edge-core) and extends TimeoutStartSec while bytes are still arriving. The parser, fan-out worker, and tests live in cyberwave_edge_core/driver_logs.py and tests/test_docker_pull_progress.py.

Startup is reported as three distinct stages. overall_progress_percent — the number a progress bar should read — spans all three as a single 0–100 that only moves forward (download 0–45, install 45–80, container 80–100), because the card renders one bar and a stage restarting its own count reads as the bar rewinding. stage_progress_percent carries the same startup as a per-stage 0–100, for a stage-aware bar to use; the alert's text names the stage on screen, so a stage that reads 100 % is one you can check against what you are being told. The bands are sized by how long each stage takes rather than by how many steps it has, so install — the stage that outlasts the download — gets a comparable share of the bar.

Downloading is driven by layers fetched rather than by bytes: docker only declares a layer's size when it starts fetching it, so a byte ratio has an unknown denominator for most of the pull and reads 100 % every time the in-flight batch drains. The layer currently downloading counts as its byte fraction, so the number still moves between layers. 100 % is reserved for a pull docker has confirmed finished. Installing — verifying and decompressing, which on a multi-GB image outlasts the download — tracks layer completion (layers_complete / layers_total). Starting covers container create → run → probe.

Local driver builds are never pulled

A driver image whose tag is local, starts with local- or ends with -local is treated as a developer build. When such an image already exists on the host, Edge Core uses it as it is: _use_local_build takes it out of the parallel pull, the startup alert records phase: "pull_skipped", and the log says Using the local build of <image>; local-only tags are not pulled. Before this rule a restart pulled the tag like any other, and because Docker Hub can carry an image with the same tag, the pull quietly replaced the build you were testing. A local-only tag that is not on the host is still pulled, and digests (@sha256:…) never count as local. The worker has its own, broader rule; see Worker image refresh policy.

Which image a twin's driver runs

The docker_image in the twin's metadata.drivers is what runs, whatever tag it carries (humble-dev-sha-<sha>, humble-0.1.3-rc.7, humble-pr-4549-sha-<sha>, a @sha256: digest, a release such as jetson-humble-0.1.23). After docker ps on the edge, the driver's own services show that reference, not a moving channel tag.

A driver image can declare a stack of sibling images in its io.cyberwave.driver.compose.* label (the Go2 declares go2-ros2-driver, ros2-slam and ros2-nav2). Edge Core resolves that graph as follows:

  1. Own services. A service of the same repository as the twin's image (plant and bridges for the Go2) runs the twin's reference exactly. The label names it <repo>:${CW_CHANNEL_TAG}, which used to resolve to the channel the image was built in, so a twin on humble-dev-sha-<sha> ran humble-dev. See _run_the_twin_image_in_its_own_services in cyberwave_edge_core/startup.py.
  2. Siblings of a pinned graph. CI pins every sibling to an immutable -sha- tag and marks it x-cyberwave.pinned_from (pin-driver-compose-images, .github/scripts/pin_driver_compose.py). Edge Core runs those tags as written. That is what keeps driver and siblings on one release.
  3. Siblings of an unpinned graph. If the twin's image is a release tag (<distro>-X.Y.Z, optionally jetson- led), the ${CW_CHANNEL_TAG} siblings are requested at that same tag; otherwise they follow the image's io.cyberwave.image.channel-tag label. See _resolve_channel_tag in cyberwave_edge_core/driver_selection.py. PR re-tags (humble-pr-…), dev prereleases (humble-0.1.3-dev.5) and moving tags are not release tags and keep following the label. Siblings keep their own version counters, so ros2-slam:jetson-humble-0.1.23 is generally not a published image: step 4 refuses this case rather than letting those services fail to pull. The simulator reads the same graph through get_image_declared_services but has no step 4, so a release-tagged twin there fails to pull those siblings instead of being refused.
  4. No proof of one release, no start. A sibling that is still declared by placeholder after step 2 is whatever its channel resolves to, so the stack may mix releases, or name an image that does not exist. Moving tags (humble-dev) and version-shaped tags that came from step 3 both count: the check looks at how the graph named the sibling, not at the shape of the tag. Edge Core refuses to start that twin: the log has Twin '<name>' refused: driver image … does not pin its sibling services to a release … : slam=<image>, and a driver_start_failure alert titled Driver stack is not pinned to one release names the services. Only drivers with sibling images are affected, and only a twin that names one build (a release, a -sha- tag, a digest): a twin on a moving tag such as humble asks for a channel, its siblings follow the same channel, and it starts as before.

Ways to run such a twin: use a driver release whose images are pinned; pin each named service in the twin's service_image_overrides (a tag or a @sha256: digest; a service pinned there counts as pinned); or set CYBERWAVE_DRIVER_CHANNEL_TAG on the edge to pin the whole graph, driver services included, to one train (an operator override; it disables steps 1 and 4).

service_image_overrides is applied last and always wins. An image without a registry namespace (a bare local build name) is never refused, because there is nothing to tell its siblings from third-party images such as redis.

Tests: tests/test_image_declared_topology.py and tests/test_resolve_twin_driver_specs.py.

When docker create times out

docker create runs with a timeout (CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS, 30 s by default), but the timeout only kills the CLI. A busy Docker daemon usually goes on to finish the create, which used to leave a container in Created that Edge Core had never registered and therefore never retried. Edge Core now waits up to 10 more seconds for the container to appear (_wait_for_container in docker_launch.py). If it does, the launch carries on as normal, with a warning in the log; only if it does not is the attempt failed with docker_create_timeout.

Model Manager (ML model cache)

Edge Core includes a ModelManager that resolves ML model weights into a local cache before starting the worker container. It is designed for both online deployments and air-gapped sites: it prefers fresh weights from Cyberwave when the network is available, falls back to upstream public mirrors, and finally to whatever is already on disk.

Cache location:

Platform Default path
All ~/.cyberwave/models/

Override with CYBERWAVE_EDGE_CONFIG_DIR.

Cache layout:

<cache_dir>/
├── manifest.json            # index of all cached models
├── yolov8n/
│   ├── yolov8n.pt           # weight file
│   └── metadata.json        # checksum, runtime, source URL, upstream URL
└── background-subtraction/
    └── ...

Model requirements discovery: Edge Core scans *.py files in ~/.cyberwave/workers/ for cw.models.load(...) calls to determine which weights to ensure.

~/.cyberwave/models/ and ~/.cyberwave/workers/ are created eagerly on Edge Core startup (even before any worker runs), with ownership matching the invoking user, so operators can drop pre-staged weights into ~/.cyberwave/models/{model_id}/ from a regular shell.

On systemd deployments where Edge Core itself runs as root, every per-model staging directory, downloaded weight, sidecar metadata.json, and the top-level manifest.json is also chowned to the invoking host user at write time — not just at startup. This keeps the cache both readable and writable from inside the worker container (which launches with --user $UID:$GID); without it, cw.models.load crashes with PermissionError: [Errno 13] on iterdir when traversing a root-owned cache_dir/{model_id}/. Writability matters because the SDK's UltralyticsRuntime sandboxes lazy secondary-asset downloads (e.g. the MobileCLIP text encoder for YOLOE / YOLO-World text prompts) into the weight directory. The wiring lives in ModelManager(owner_uid_gid=resolve_config_owner_uid_gid()).

Resolution order

For each required model, ensure_model(model_id) runs the following steps:

  1. Reconcile disk. If cache_dir/{model_id}/ already contains a weight file (with or without a sidecar), it is registered in the manifest. A missing sidecar is generated from the on-disk file (with a freshly computed SHA-256) and tagged downloaded_from: prestaged. This is how an operator pre-stages weights from a USB stick on an air-gapped site.

  2. Verify cache integrity. If the local file's SHA-256 matches the manifest checksum, the cache is intact.

  3. Pre-staged short-circuit. When an intact entry is tagged downloaded_from: prestaged, Edge Core returns it without ever contacting the catalog. Pre-staged files are operator-curated truth; to force a re-download, evict the model directory.

  4. Best-effort catalog probe. For non-prestaged intact entries, Edge Core does a short-timeout GET /api/v1/mlmodels/... to compare checksums.

    • Catalog unreachable, no checksum, or matching checksum → return the cached file (no download).
    • Catalog returns a different checksum → fall through to the download path.
  5. Download. Sources are tried in priority order:

    1. Cyberwave-hosted signed URL from GET /api/v1/mlmodels/{uuid}/weights — used for checkpoints we have uploaded to our private GCS bucket (e.g. internally trained or mirrored models). Authenticated, served from infrastructure we control.
    2. Upstream weights URL from the catalog entry (download_url / metadata.upstream_weights_url) — used for community checkpoints we did not mirror.
    3. Runtime-managed download as a final fallback: when neither URL is set but the catalog entry's edge_runtime ships its own weight resolver (today: ultralytics against its GitHub releases hub), Edge Core invokes it in a subprocess and caches the resulting file as downloaded_from: runtime_managed. This is what lets .pt catalog entries (YOLO26, YOLOE-26) work without a mirrored upstream URL. Requires the runtime to be installed on the edge host — see ML runtime extras.

    The first source that yields a checksum-verified file wins. The sidecar records downloaded_from (artifact_url / download_url / runtime_managed / prestaged), source_url (the public URL we fetched, or null for artifact and runtime-managed downloads — neither is a re-fetchable URL we want to persist), and upstream_url (provenance).

  6. Fail-soft. If every download attempt fails and the cached file is intact, Edge Core returns the cached path with a warning. This keeps workers running across transient network failures and on permanently air-gapped sites. If the cache is empty or corrupt, RuntimeError is raised.

Cache integrity: SHA-256 checksums are verified on every ensure_model call when a checksum is recorded — on cold start, after every download, during disk reconciliation, and on every warm-cache hit. There is no shortcut. For multi-gigabyte checkpoints this is the dominant cost of the call, but ensure_model only runs when a worker file changes (rare) or the worker container restarts; in exchange we get unconditional bit-rot detection. A checksum mismatch on a downloaded artifact triggers a re-download attempt; a download whose checksum does not match the catalog is rejected and the partial file removed.

Pre-staging weights for air-gapped deployments

On a site without internet access, an operator can place weights directly into the cache:

mkdir -p ~/.cyberwave/models/yolov8n
cp /usb-stick/yolov8n.pt ~/.cyberwave/models/yolov8n/

Edge Core picks up the file on the next ensure_model("yolov8n") call, computes a SHA-256, and writes a sidecar metadata.json so subsequent runs are deterministic. The runtime field is inferred from the file extension (.pt → ultralytics, .onnx → onnxruntime, .engine/.trt → tensorrt, .tflite → tflite, .pth → torch, .xml → opencv); provide a hand-written metadata.json to override.

Updating in place. Operators can drop a new build into the same directory and Edge Core will detect the change on the next call:

cp /usb-stick/yolov8n-v2.pt ~/.cyberwave/models/yolov8n/yolov8n.pt

The mismatch between the on-disk SHA-256 and the manifest checksum triggers a re-stamp (not a re-download), provided the sidecar still records downloaded_from: prestaged. This keeps offline edges functional across model upgrades. Files that were previously downloaded by Edge Core keep the corruption-detection semantics — bit-rot still triggers a re-download attempt rather than being silently accepted.

Pre-staged files are never auto-overwritten by catalog updates. Once a file lives under cache_dir/{model_id}/ with a downloaded_from: prestaged sidecar, Edge Core treats it as the source of truth and skips the catalog probe entirely. To force a re-download from the Cyberwave catalog, evict the model:

rm -rf ~/.cyberwave/models/yolov8n

Provide a hand-written metadata.json (with filename, checksum_sha256, and runtime) when there are multiple weight files in the directory or when corruption detection should compare against a known-good hash.

Self-healing the model cache

A failed download leaves an empty cache_dir/{model_id}/ directory behind — the mkdir on the download path runs before the network fetch, and any error in the downloader (HTTP timeout, missing upstream weight, runtime self-download crash) skips the cleanup. Without intervention, the SDK in the worker container would later route that directory into torch.load and crash with IsADirectoryError on every restart.

Edge Core handles this in three layers:

  • Per-download cleanup (conservative). If _download_model or _download_runtime_managed raises, the staging directory is removed before the exception propagates, provided it contains only an .dl_*.part partial-download file, a metadata.json sidecar, or nothing at all. Operator-staged content blocks the prune.
  • Startup sweep (conservative). ModelManager() walks cache_dir/ once at construction time and prunes any per-model subdirectory that is not tracked in the manifest and contains only orphan cruft. This catches hosts that were already wedged before this code shipped. Operator-staged content blocks the prune.
  • evict_model('foo') (decisive). When there is no manifest entry but cache_dir/foo/ exists on disk, the directory is removed unconditionally and True is returned. This is the recovery hook for a remote eviction API to call without shell access to the host; an explicit evict is treated as a stronger operator-intent signal than the conservative auto-sweeps.

For the two conservative layers, a directory containing an unexpected file (an operator's README.md, a hand-staged weight file with an unusual name, a sub-directory) is preserved untouched — "human always wins". Use evict_model explicitly when you mean to clear the slot regardless of contents.

ML runtime extras

The default pip install cyberwave-edge-core keeps the runtime surface lean — no Torch, no Ultralytics, no ONNX. That's enough for workflows that resolve weights through a backend-hosted URL or pre-staged files. Catalog entries that rely on the runtime-managed download fallback (step 5.3 above) need the matching runtime installed on the edge host:

# Pull in Ultralytics so YOLO/YOLOE .pt entries can self-download
pip install 'cyberwave-edge-core[ml]'

# Other available extras (mirror the cyberwave SDK extras 1:1):
pip install 'cyberwave-edge-core[ml-onnx]'       # onnxruntime
pip install 'cyberwave-edge-core[ml-tflite]'     # tflite-runtime
pip install 'cyberwave-edge-core[ml-wakeword]'   # openwakeword + onnxruntime
pip install 'cyberwave-edge-core[ml-stt]'         # pywhispercpp (Whisper.cpp GGML)
pip install 'cyberwave-edge-core[ml-stt-faster]'  # faster-whisper (CTranslate2)
pip install 'cyberwave-edge-core[ml-audio]'      # silero-vad + torchaudio (Audio Assistant)
pip install 'cyberwave-edge-core[fuzzy-match]'   # rapidfuzz (Fuzzy Matcher)
pip install 'cyberwave-edge-core[microphone]'    # host mic capture (Audio Track trigger)
pip install 'cyberwave-edge-core[ml-all-vision]'  # YOLO / ONNX / TFLite only
pip install 'cyberwave-edge-core[ml-all-audio]'   # all speech extras (avoid on Pi — pick ml-stt / ml-stt-faster per model)
pip install 'cyberwave-edge-core[ml-all]'         # CI/dev fat bundle only

When a required runtime is missing, the resolver raises a RuntimeError that spells out the three workarounds: drop the weight file into ~/.cyberwave/models/{model_id}/, upload to /api/v1/mlmodels/{uuid}/weights, or set metadata.download_url on the catalog entry.

On a frozen PyInstaller binary (cyberwave edge install --channel dev|staging), sys.executable is the edge-core CLI wrapper rather than a Python interpreter, so the runtime-managed downloaders cannot spawn the subprocess they need. In that case ensure_model raises _RuntimeManagedDeferred, which ensure_models treats as non-fatal: the model is omitted from last_ensure_failures and the worker container — which bundles the runtime — resolves the weight on its first inference call. This prevents a misleading model_download_failure alert from firing while the workflow runs fine (CYB-2103).

Worker container

Edge Core manages one ML worker container per edge device (container name: cyberwave-worker-{env_uuid[:8]}). The worker container runs Python worker scripts from the local workers directory and has access to cached model weights.

Worker directory layout

Place worker scripts in {config_dir}/workers/ (default: ~/.cyberwave/workers/):

~/.cyberwave/
├── workers/
│   ├── detect_people.py        # Custom worker
│   └── cyberwave.yml           # Optional: list model requirements
└── models/                     # Auto-managed model cache
    ├── manifest.json
    └── yolov8n/
        └── yolov8n.pt

cyberwave.yml

Optionally declare model requirements so Edge Core can pre-download them before starting the worker container:

models:
  - yolov8n
  - background-subtraction

Edge Core also auto-detects models by scanning cw.models.load("...") calls in worker Python files.

Worker container environment variables

Variable Value
CYBERWAVE_API_KEY Injected from credentials
CYBERWAVE_ENVIRONMENT_UUID Active environment UUID
CYBERWAVE_TWIN_UUIDS Comma-separated twin UUIDs in environment
CYBERWAVE_DATA_BACKEND zenoh
ZENOH_CONNECT Set when a Zenoh router is configured
ZENOH_SHARED_MEMORY false by default (opt-in; requires --ipc=host)

File watching and hot-reload

Edge Core monitors {config_dir}/workers/ every reconcile cycle (~15 seconds). When .py files are added, removed, or modified, Edge Core automatically:

  1. Re-scans model requirements.
  2. Pre-downloads any missing models.
  3. Restarts the worker container with the updated files.

A minimum cool-down of 10 seconds between successive automatic restarts prevents rapid churn when files are written incrementally (e.g. by rsync or scp).

Workflow-driven worker lifecycle

The worker container is brought up and torn down based on whether any active workflows are currently synced for the connected twins:

  • Startup: after pulling worker files from the backend (step 8), Edge Core inspects {config_dir}/workers/. If at least one wf_*.py file is present, the worker container is started; otherwise it is left down — and the cyberwaveos/edge-ml-worker image is not pulled.
  • Periodic reconcile: every ~5 minutes (configurable via CYBERWAVE_WORKER_SYNC_INTERVAL_LOOPS), Edge Core resyncs worker files from the backend. If a workflow was activated mid-run and new files appeared, the worker container is started. If every workflow was deactivated and the directory is now empty, the container is stopped. Both calls are idempotent.
  • Immediate reconcile on activate: when a run_on_edge workflow is activated (UI, CLI, or API), the backend publishes a sync_workflows command on cyberwave/twin/{twin_uuid}/command for each twin the workflow references. Edge Core runs reconcile_worker_sync right away so the new wf_*.py lands within seconds instead of up to one periodic interval. When the worker container is already running, Edge Core also bounces it via WorkerManager.restart so the new modules are actually imported — start() is a no-op for a running container, so without this step activation would only take effect after the next file-watch tick (or a manual cyberwave worker restart). The bounce is skipped when the container itself started after the newest change in {config_dir}/workers/ — it already imported those files on the way up, so restarting would only discard a boot in progress and delay the workflow going live. Anything that cannot be determined (a failed docker inspect, an unreadable workers directory) falls back to bouncing. Failures fall back to the periodic reconcile; the MQTT nudge is best-effort, not a correctness guarantee.
  • Immediate reconcile on remove: when a run_on_edge workflow is deactivated, soft-deleted, or has its run_on_edge flag flipped off, the backend publishes a surgical remove_workflow_worker command on the same twin command topic, naming the specific worker filename(s) to unlink. Edge Core unlinks the matching wf_*.py from {config_dir}/workers/ and immediately reconciles the worker container lifecycle, so deactivation propagates within seconds rather than waiting up to one periodic interval (~5 minutes) plus the bulk-sync two-strikes cleanup cycle. For partial deactivations (sibling workflows still active), the same in-place restart fires so the removed worker is actually unloaded. The same nudge also covers workflows soft-deleted as part of a workspace or environment teardown. The periodic reconcile remains the correctness backstop if the MQTT publish is dropped.
  • MQTT command dedupe: both sync_workflows and remove_workflow_worker carry a request_id; Edge Core keeps a bounded FIFO of the last 1024 IDs it has handled per process and drops repeats. QoS retries and broker re-publishes therefore don't spawn duplicate sync threads or repeat unlink operations.
  • Sync errors: if a sync cycle reports any errors, the lifecycle reconcile is skipped to avoid churning a healthy worker on transient API failures, and stale-file cleanup is skipped too. A twin whose fetch failed reports no workflows, which is indistinguishable from a genuine deactivation, so deleting on that picture would let an outage, an expired token or a DNS blip spanning two sync periods unlink every wf_*.py on the device. Deletion needs a complete picture; the next successful sync re-evaluates state.

How this is tested

Reconcile branching, watcher debouncing, and MQTT command handling are unit-tested with mocked Docker in tests/test_worker_lifecycle_reconcile.py, tests/test_worker_watcher.py, tests/test_worker_health.py, and tests/test_startup_remove_workflow_worker.py. ResourceLimits argv assembly is covered in tests/test_worker_health.py (TestResourceLimits).

Tier 1 — scripted lifecycle integration (tests/integration/test_worker_lifecycle_docker.py, marker: docker): 14 deterministic phases exercise the real Docker daemon — cold start, add/remove worker files, restart-in-place, surgical remove, edge-core process restart, cooldown enforcement, image pull policy, request_id dedupe for both remove_workflow_worker and sync_workflows, health-monitor record_stop wiring, spontaneous-exit detection, hot-reload through WorkerWatcher.force_restart, and a full activate→deactivate→reactivate operator cycle. The suite auto-skips when Docker is unavailable and uses a session-built cyberwave-worker-stub image so it never pulls the real cyberwaveos/edge-ml-worker.

Tier 2 — Hypothesis chaos (tests/integration/test_worker_lifecycle_chaos.py, marker: docker): a RuleBasedStateMachine drives the same surface (WorkerManager, reconcile_worker_lifecycle, _run_remove_workflow_worker, WorkerWatcher) with randomized operation interleavings against the real Docker daemon, asserting eight cross-cutting invariants after every step (state consistency, no spurious docker rm, container-name stability, request_id dedupe, immutable-tag pull-at-most-once, record_stop count matches running→exited transitions, cooldown respected, no orphan containers). PR CI runs with max_examples=20, stateful_step_count=30, derandomize=True (deterministic, ≤ 3 min); staging also runs a lighter chaos slice (max_examples=5, stateful_step_count=15). The nightly workflow (.github/workflows/edge-core-nightly-chaos.yml) mirrors the PR test_edge_core job (Tier 1 Docker integration, Tier 3 + Tier 3-ML shell E2E against a live backend) and adds Tier 2 chaos with max_examples=200, stateful_step_count=100 and a random seed; it files a GitHub issue on failure without blocking PRs. Monotonic timing for cooldown and health windows goes through cyberwave_edge_core._clock.now_monotonic() so tests can advance time without sleeping.

Tier 3 — Full E2E against a live backend (cyberwave-edge-nodes/test_worker_lifecycle_e2e.sh): closes the gaps left by Tier 1/2 by exercising the complete workflow activation pipeline: creates real backend objects (project → environment → twin → workflow with manual-trigger + send-MQTT nodes) via the HTTP API, activates the workflow, and asserts that the sync_workflows MQTT message is delivered to a running edge-core, that edge-core fetches worker_source from the /edge-sync endpoint, writes the resulting wf_*.py file, and starts the worker container. Deactivation is then exercised and the container stop is asserted. Runs after test_edge_core_and_cli.sh in the same CI job so the backend and broker are already healthy. The test is self-contained: it builds a minimal cyberwave-worker-stub:it (Alpine sleep infinity) and a cyberwave-edge-test image, and tears both down on exit.

Tier 3-ML — Model download and inference E2E (cyberwave-edge-nodes/test_ml_inference_e2e.sh): extends the Tier 3 test with two focused phases. Phase A creates a camera_frame → call_model(yolov8n.onnx) workflow, activates it, and asserts that edge-core pre-downloads yolov8n.onnx (~6-12 MB, YOLOv8 Nano ONNX from the Cyberwave CDN) via ModelManager.ensure_models() before starting the worker container. Phase B starts cyberwaveos/edge-ml-worker:local and runs onnxruntime inference on a synthetic 640×640 image, asserting clean exit and a valid 3-D output tensor. Requires cyberwaveos/edge-ml-worker:local to be pre-built (see the script header for the docker buildx build command); the CI step skips gracefully when the image is not available.

Worker image refresh policy

WorkerManager._ensure_image_pulled decides whether to issue docker pull before each worker (re)start:

Tag basename (with optional -gpu/-cpu/-arch suffix) Mutability Pull behaviour
latest, dev, staging, nightly, edge, main, master Mutable Pull every time, even when the image is already present locally. If the registry is unreachable but a local copy exists, fall back to the local copy and warn.
Anything else (v1.2.3, dated build IDs, @sha256:…, local) Immutable Skip the pull when the image is already present locally; only pull when missing.

This avoids the previous failure mode where a stale cyberwaveos/edge-ml-worker:dev-gpu image stayed cached after a developer pushed a new build — operators no longer need to remember to docker rmi before restarting the worker. Immutable tags keep the original fast-path so versioned production deployments are not slowed down by an extra round-trip to the registry on every restart.

local counts as immutable because nothing publishes it: a :local image is whatever the last local docker build produced, so a pull can only fail — a wasted round-trip, or a connect timeout on an offline machine. A :local tag with no local copy is still pulled, like any other missing image.

Driver images follow a narrower rule. They are pulled on every start unless the tag is local, local-* or *-local and the image is already on the host (see Local driver builds are never pulled). The reason is the same: a local build must survive a restart.

Pinning a custom worker image (CYBERWAVE_WORKER_IMAGE)

resolve_worker_image() consults CYBERWAVE_WORKER_IMAGE before falling back to the CYBERWAVE_ENVIRONMENT-derived tag. This is the worker-side counterpart to the driver_overrides field in credentials.json for camera/asset drivers, and is the recommended way to run a locally-built or hot-patched worker image without re-pushing to the registry.

Two ways to set it (both honoured by get_runtime_env_var):

  • Operator config (preferred for dev hosts, no sudo) — add to the envs block of ~/.cyberwave/credentials.json:

    {
      "envs": {
        "CYBERWAVE_ENVIRONMENT": "dev",
        "CYBERWAVE_WORKER_IMAGE": "cyberwaveos/edge-ml-worker:local"
      }
    }
    
  • Systemd dropin (preferred for managed deployments) — sudo systemctl edit cyberwave-edge-core:

    [Service]
    Environment=CYBERWAVE_WORKER_IMAGE=cyberwaveos/edge-ml-worker:local
    

Either way, restart edge-core (sudo systemctl restart cyberwave-edge-core) to reload the resolver.

Use the :local tag (no -gpu/-cpu/-jetson/-hailo suffix) — same convention as the camera-driver :local tag. _run_container auto-appends the accelerator suffix (-jetson on Jetson hosts, -gpu on discrete-NVIDIA hosts, -hailo on Pi 5 + AI HAT+) for cyberwaveos/edge-ml-worker:* overrides, so the operator only commits and pins one tag.

Typical hot-fix loop (mirroring the camera-driver :local tag pattern):

# 1. Hot-patch the running container (e.g. swap an SDK runtime file).
WORKER=cyberwave-worker-<fingerprint>
docker cp /path/to/patched/file.py "$WORKER:/usr/local/lib/python3.12/dist-packages/cyberwave/.../file.py"
docker exec -u root "$WORKER" rm /usr/local/.../file.cpython-312-x86_64-linux-gnu.so

# 2. Snapshot the patched container as a local-only tag (and an alias on the
#    accelerator-suffixed tag matching the host, so the same image is used
#    regardless of which path `_run_container` resolves to on this host).
docker commit "$WORKER" cyberwaveos/edge-ml-worker:local
docker tag    cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu
# On Jetson hosts, alias -jetson instead of -gpu:
# docker tag  cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-jetson

# 3. Tell edge-core to use it (see the two options above), then restart it.
#    Pull will fail (registry has no :local) and `_ensure_image_pulled`
#    falls back to the locally-present image.
sudo systemctl restart cyberwave-edge-core

The patched image survives every docker rm/docker run cycle from edge-core's reconcile loop. To revert, remove the env var (or systemctl revert cyberwave-edge-core if you used the dropin) and docker rmi cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu (or …local-jetson on Jetson hosts).

When overriding to a registry outside cyberwaveos/edge-ml-worker:* (e.g. an internal mirror), include the accelerator suffix (-gpu / -jetson / -hailo) yourself if you need accelerator access — the auto-suffix path in _run_container only triggers for the canonical cyberwaveos/edge-ml-worker: prefix.

Worker startup probe

After docker run --detach, Edge Core polls the container status for up to 30 seconds (configurable via CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS) waiting for it to reach running state. The probe uses docker inspect to check the full container state, including restart count:

  • running: success — container is up.
  • exited/dead with no prior restarts: definitive failure — an alert is sent to all linked twins.
  • exited with RestartCount > 0: Docker's restart policy (--restart=unless-stopped) is active — the probe keeps waiting.
  • created/restarting/other transient states: Docker is still setting up the container — the probe keeps waiting.
  • Probe window elapsed: the container may still come up (Docker keeps trying via its restart policy), so Edge Core returns success and sends a worker_start_failure alert for visibility. The health monitor and periodic reconcile loop track the eventual outcome.

This resilient approach ensures that slow container startups on resource-constrained devices (e.g. Raspberry Pi) do not block the rest of Edge Core's boot sequence.

Worker readiness

A running container is not yet a usable worker. The process inside still has to import your worker modules (downloading model weights the first time), connect to MQTT, subscribe its hooks and run warm-up inference — minutes on a cold device.

The worker reports each of those milestones to /run/cyberwave/worker-status.json, which Edge Core mounts into the container and reads once per reconcile cycle (every 15 s). The worker_starting alert stays open across the whole sequence, so the dashboard shows Loading models or Warming up models — 2 of 5 models instead of going quiet the moment the container starts, and a workflow run dispatched to the device explains what it is waiting for rather than appearing stalled.

  • ready: the alert resolves.
  • Container exits during startup: the alert reports the failure and the phase it reached.
  • CYBERWAVE_WORKER_READY_TIMEOUT_SECONDS elapsed (default 15 minutes): the alert reports that the worker took too long to load its models.
  • No status file within 60 seconds: the worker image predates status reporting, so Edge Core falls back to treating a running container as started.

The readiness timeout is measured from the container's own start timestamp rather than from when Edge Core began watching, so a long image pull does not eat into the worker's own startup budget and a container restart resets the clock by itself.

The progress card does not survive an Edge Core restart, though. A graceful restart stops the worker container first, so the next start opens a fresh card. But if Edge Core is killed while the worker container keeps running — an out-of-memory kill, a watchdog reset, a crash loop — the card is resolved on the next boot and is not re-opened for that container, and the worker finishes loading with nothing on screen.

cat /run/cyberwave/worker-status.json on the device shows the current phase directly. The file lives on tmpfs, so it costs no flash writes and does not survive a reboot.

Worker health monitoring

Edge Core continuously monitors the worker container for spontaneous exits and crash loops:

  • Restart accounting: every restart is recorded with a timestamp and reason.

  • Sliding-window rate limiting: if more than 5 restarts occur within 5 minutes, the circuit-breaker trips and automatic restarts are suppressed. The breaker resets automatically once the window clears.

  • Spontaneous exit detection: if the container exits without a deliberate restart, a warning is logged so operators can investigate. The detector recognises deliberate stops via two channels:

    • Callers that hold the monitored WorkerManager (e.g. WorkerManager.stop) notify it via record_stop(). The steady-state workflow deactivation path (reconcile_worker_lifecycle) reuses the watcher's monitored manager so this channel covers it directly.
    • For cross-instance stops driven by managers without the shared monitor (the "Restart edge core" flow's _stop_worker_container_for_restart, and cold-boot MQTT command handlers that fire before the watcher's first tick), the monitor consults an "expected to be running" probe that returns False when either the workers directory is empty or an edge-core restart is in flight. Both cases downgrade the log to INFO.

    The "may indicate a crash loop" wording is reserved for exits that look unexpected — real crashes, OOM kills, or manual docker stop while workflows are active.

Use cyberwave-edge-core worker health to inspect the full restart history and circuit-breaker state.

Resource limits

You can constrain the worker container's CPU and memory usage by setting CYBERWAVE_WORKER_CPU_QUOTA_PERCENT and CYBERWAVE_WORKER_MEMORY_MB environment variables on the edge host (both optional). When set, Edge Core passes the corresponding --cpu-quota, --cpu-period, and --memory flags to docker run.

GPU memory fraction can be limited via CYBERWAVE_GPU_MEM_FRACTION (a float between 0 and 1); this is passed as an env var into the worker container.

High availability and watchdog

Edge Core includes several layers of resilience for edge devices running ML inference workloads (e.g. Raspberry Pi 4 with YOLO). These features prevent the orchestrator from dying under heavy CPU or memory load and ensure automatic recovery when failures occur.

There are four independent watchdog/supervision layers, each covering a different failure mode:

Layer What it watches Mechanism Recovery action
Hardware watchdog Kernel hang, total board lockup /dev/watchdog ping SoC resets the board
Systemd watchdog Orchestrator process hang/exit Type=notify + WatchdogSec systemd restarts the service
Worker container ML worker container health WorkerHealthMonitor + circuit breaker Docker restart / circuit-break
Driver containers Per-driver containers that flap, exit or disappear reconcile_driver_* in startup.py restart suppression on flap; revival of clean exits, and of removed containers after a grace period

Note: these are infrastructure watchdogs. They are distinct from the in-process command/teleop watchdogs in cyberwave-edge-nodes and cyberwave-edge-runtime, which stop driver motion if no commands arrive within a window — the term "watchdog" is overloaded.

Systemd watchdog

The systemd unit generated by cyberwave edge install configures WatchdogSec=60 and Type=notify. Edge Core pings the systemd watchdog every reconcile cycle (~15 s). If the process hangs or dies, systemd restarts it automatically. The unit uses Restart=always (which is the systemd equivalent of Docker's unless-stopped — systemd skips the restart when the operator runs systemctl stop). always is preferred over on-failure for robustness: both restart on SIGKILL/OOM, but only always also restarts on an accidental exit(0) from a buggy code path.

Startup readiness uses a two-phase handoff (ProcessWatchdog.mark_ready() → ProcessWatchdog.start_pinging()): READY=1 is sent unconditionally as soon as the blocking boot work (driver images pulled, MQTT connected, twin sync reconciled) is done, while the periodic WATCHDOG=1 pings — which sd_notify(3) restricts to the $WATCHDOG_PID process — are driven from the runtime loop in the same process. The split is what makes the standalone PyInstaller binary work under Type=notify: the --onefile bootloader holds MainPID (and therefore $WATCHDOG_PID) while the actual Python app runs in a forked child whose PID differs.

During long-running startup operations (driver image pulls and ML worker preparation, including model downloads and the worker image pull), Edge Core sends EXTEND_TIMEOUT_USEC notifications via sd_notify every 5 seconds. This extends the systemd start timeout while preparation is active, including periods without pull progress events, so TimeoutStartSec does not kill the service during a slow download, but not without limit: extensions stop when the startup budget (30 minutes by default) has passed since the watchdog was created, and they stop at once on SIGTERM (see Bounded startup and shutdown below). A STATUS= notification is also sent so that systemctl status shows which images are being pulled. On SIGTERM, startup checks stop advancing to new drivers or workers. An in-flight Docker operation may still finish or reach systemd’s normal stop timeout; it cannot prolong shutdown with startup heartbeats.

Because the watchdog pings come from the runtime loop, anything that blocks the loop for a minute gets the service killed, and an SDK REST call made without a timeout could do exactly that when the backend never answered. Before the runtime loop starts, main.py therefore calls bound_sdk_request_timeouts(), which patches the generated SDK client so that every request without its own timeout gets one of 10 s (SDK_REQUEST_TIMEOUT_SECONDS). Requests that already pass a timeout keep it, the patch is applied once however often it is called, and boot-time calls made before it are not affected. The bound applies to each attempt, not to the call: urllib3's default policy retries a timed-out GET three times, so one GET can still take about 40 s. Nor does it bound a sequence of calls: a driver revival pass makes one GET for the twin list and up to two per linked twin, one after another. The trade-off is deliberate: a legitimately slow REST call from the runtime loop now fails and is retried on a later cycle, instead of taking the whole service down with it.

Bounded startup and shutdown

Startup keeps Type=notify readiness and a base TimeoutStartSec=900. The pull heartbeat above can extend it, but only up to the startup budget, counted from the watchdog's creation, so a startup that never succeeds cannot stay "activating" forever. The budget is 30 minutes; an edge whose first start pulls several gigabytes over a slow link can raise it with CYBERWAVE_STARTUP_TIMEOUT_MINUTES, a positive whole number of minutes (any other value is ignored with a warning). Set it in the envs of credentials.json: the service reads it from there, and so does the unit generator, which derives JobTimeoutSec from it and deliberately ignores the shell it runs in, since the service never sees that shell. After changing it, run sudo cyberwave edge install again, or let the next package upgrade regenerate the unit, before restarting the service. SIGTERM calls ProcessWatchdog.begin_shutdown(): it sends STOPPING=1 and refuses every later extension, READY=1 and watchdog ping, even when the signal arrives in the middle of the startup checks, which then exit through the normal shutdown path without announcing readiness. A consequence worth knowing: /dev/watchdog is never opened after SIGTERM, so a stop can never arm the hardware reset.

The generated unit also bounds the other end. TimeoutStopSec=90 gives the orchestrator 90 seconds to stop before systemd kills what is left in the service's control group (KillMode=control-group, SendSIGKILL=yes), and JobTimeoutSec in the [Unit] section bounds the start job as a whole, including time spent waiting behind its dependencies, five minutes beyond the startup budget (35min by default). Killing the orchestrator does not stop Docker containers by itself, so CLI uninstall checks separately that the service stopped and removes the containers before deleting configuration.

The CLI side matches: on Linux, cyberwave edge start, restart and install submit the systemd job without waiting for readiness and print the commands to follow it, while stop and uninstall wait up to 120 seconds for the service to stop and, if they cannot confirm it, report the failure and keep the configuration. Deploy both packages to get both halves. The media and lifecycle investigation has recovery steps, limits and verification commands.

Hardware watchdog (Raspberry Pi)

On Linux devices with /dev/watchdog (all Raspberry Pi models), Edge Core optionally drives the hardware watchdog timer. If the edge-core process stops pinging for longer than the hardware timeout, the SoC resets the board. This is a last-resort safety net for kernel-level hangs or total process death.

Edge Core asks for twice the unit's WatchdogSec, and at least 120 seconds, so a hung process is first restarted by systemd and only a hang that outlives that reboots the board. Override it with CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS. A driver may round or clamp the request, and some refuse it outright and keep their own timeout: a Raspberry Pi's bcm2835_wdt tops out at 15 seconds. The log line Hardware watchdog enabled (requested timeout=…, effective=…) shows the timeout actually running, and a warning follows when it is not above WatchdogSec.

Disable with CYBERWAVE_HARDWARE_WATCHDOG=false.

OOM protection

Edge Core lowers its OOM score (oom_score_adj=-800) at startup so the Linux OOM killer preferentially terminates the heavier ML worker container (which Docker restarts automatically via --restart unless-stopped) instead of the orchestrator. The systemd unit reinforces this with OOMScoreAdjust=-800.

Auto-detected memory limits

On devices with ≤4 GB total RAM, Edge Core automatically applies a memory limit to the worker container, reserving ~25% of total memory for the OS and edge-core. This prevents the worker from consuming all available memory and triggering the OOM killer. Explicit CYBERWAVE_WORKER_MEMORY_MB values override the auto-detected limit. Disable auto-detection with CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT=false.

Resource monitoring

SystemResourceMonitor (resource_monitor.py) watches four host gauges: memory in use, from /proc/meminfo; CPU load, the busy share of all cores between two readings of /proc/stat; disk usage of the filesystem that holds Docker's data root; and the temperature of the hottest CPU thermal zone. The runtime loop calls check() on every tick, but the monitor reads the host at most every 30 s (MONITOR_CHECK_INTERVAL_SECONDS) and otherwise hands back its last snapshot, so the CPU figure is a 30 s average and is absent from the first snapshot. Memory at 85 % or a temperature of 75 °C logs Resource pressure elevated at INFO. Memory at 92 % or 82 °C logs CRITICAL resource pressure as a warning and increments consecutive_critical, which drops back to zero on the first reading below both critical thresholds. Each message is rate-limited to one a minute.

CPU, memory and disk also raise an alert when they saturate. reconcile_resource_saturation runs after check() and looks at each new snapshot once: two snapshots in a row at or above CYBERWAVE_RESOURCE_SATURATION_PERCENT (95 % by default; an invalid value logs a warning and falls back to 95) mark the resource saturated, and two in a row below a level five points lower (90 % by default) clear it, so a single spike or dip changes nothing. On the way up Edge Core runs docker stats --no-stream once to name the busiest containers, logs Edge CPU saturated: 97% (threshold 95%) for 30 s; top: cyberwave-driver-658d581b 61%, …, repeats that line every minute while the resource stays saturated, and sends an edge_cpu_saturation, edge_memory_saturation or edge_disk_saturation warning to every twin whose driver it manages. The alert is raised once per crossing, so the percentage in its description does not defeat the backend's dedupe; its metadata carries resource, percent and threshold. A twin the alert could not reach is retried on each later snapshot with the same content until it is delivered or the saturation clears. On the way down the alerts are resolved, and a resolve that fails is retried on the next snapshot. While the backend is unreachable, no alert is raised or resolved: the first call that fails pauses every cloud call of Edge Core, and what is left waits for a snapshot after the pause. The saturation state lives in memory, so a new process cannot know what the last one raised: when the runtime loop starts it treats every resource as possibly alerted, resolves those alerts on every managed twin at the resource's first snapshot below the clear level, and lets a resource that saturates first raise again, which the backend folds into the alert still active. An edge that manages no driver only logs. Temperature keeps its own thresholds and raises nothing.

Board power draw is sampled separately, from sysfs (Jetson INA3221 in_power0_input, generic hwmon/power1_input, or power_supply/BAT*/power_now, in that order), and fresh on every reading rather than from the 30 s snapshot, because draw can change several-fold between idle and motion in under a second. It is published as power_mw by the same two transports as the other gauges and shows up as the pwr X.XW pill in the Edge details panel of the dashboard.

Set CYBERWAVE_BATTERY_WH to your battery pack's watt-hour capacity to enable a rough runtime estimate on the dashboard (e.g. 36.0 for a 36 Wh pack). The estimate is a simple linear model (battery_wh / current_watts hours); it ignores discharge curves and temperature effects and is only meant to answer "will this run another 30 minutes?".

Variable Default Description
CYBERWAVE_HARDWARE_WATCHDOG true Enable/disable the hardware watchdog (/dev/watchdog)
CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS twice WatchdogSec, at least 120 Hardware watchdog timeout requested from the driver. Keep it above WatchdogSec. See Hardware watchdog.
CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT true Enable/disable auto-detected memory limits on ≤4 GB hosts
CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS 30 Seconds to wait for the worker container to reach running state after docker run
CYBERWAVE_WORKER_READY_TIMEOUT_SECONDS 900 Seconds to wait for the worker process to finish starting (load and warm up its models) once its container is running
CYBERWAVE_WORKER_STATUS_PATH /run/cyberwave/worker-status.json Where the worker writes its startup milestones. Set by Edge Core on the worker container to match the directory it mounts; override only if /run is unavailable
CYBERWAVE_BATTERY_WH (unset) Battery pack capacity in watt-hours. Enables the battery uptime estimate in the Edge details panel of the dashboard.
CYBERWAVE_RESOURCE_SATURATION_PERCENT 95 CPU, memory and disk usage at which Edge Core raises the edge_*_saturation alerts; they clear five points lower. See Resource monitoring.
CYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T (unset) Set to 1 to use the -jetson worker image and NVIDIA runtime on a Jetson below L4T r38 (JetPack 7). See the accelerator precedence list below.

GPU support

When an NVIDIA container runtime is detected (docker info reports nvidia runtime), Edge Core adds --gpus all to the worker container's docker run command.

For the worker image, Edge Core also rewrites the tag to point at the accelerator-specific sibling. The precedence is:

  1. Jetson host on JetPack 7 (/etc/nv_tegra_release reports L4T r38 or newer, or CYBERWAVE_PLATFORM_VARIANT=jetson) → cyberwaveos/edge-ml-worker:<tag>-jetson + --runtime nvidia. Wins over the plain -gpu branch: on Jetson the nvidia-container-runtime is present too, but the amd64 -gpu blob has the wrong libcuda.so ABI (desktop/server CUDA vs Tegra CUDA).

  2. Jetson host on JetPack 6 or older (L4T r36 and below) → plain cyberwaveos/edge-ml-worker:<tag>, no GPU passthrough, with a warning. The -jetson image's CUDA 12.9 wheels cannot load against a JetPack 6 driver — cuBLAS fails on the first matmul while convolutions still succeed, so the model loads and then detects nothing. The plain tag is multi-arch, so arm64 CPU inference works on the same host. A -jetson tag pinned via CYBERWAVE_WORKER_IMAGE is demoted the same way; a pin to an image outside cyberwaveos/edge-ml-worker keeps the NVIDIA runtime, since the floor only speaks for our own build. Set CYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T=1 to force the GPU path anyway (e.g. for a hand-built JetPack 6 image).

    The choice is made when the worker container is created. A board that already has a -jetson worker running keeps it across an Edge Core upgrade — startup skips image selection entirely when the container is already up, and its --restart unless-stopped policy survives reboots. Run docker rm -f cyberwave-worker once to pick up the new selection.

  3. NVIDIA container runtime detected (not Jetson) → cyberwaveos/edge-ml-worker:<tag>-gpu + --gpus all.

  4. /dev/hailo0 present and no GPU runtime → cyberwaveos/edge-ml-worker:<tag>-hailo + device passthrough.

  5. None → CPU base image.

If the accelerator-specific tag cannot be pulled (e.g. the sibling isn't published for your channel yet), Edge Core falls back to the base tag and logs the demotion — the worker still comes up, just on CPU.

Driver GPU passthrough is opt-in via asset metadata. When a driver's config includes "prefer_gpu": true and the host has:

  1. NVIDIA container runtime available (docker info reports nvidia), and
  2. nvidia set as the default runtime in /etc/docker/daemon.json

…Edge Core passes --gpus to the driver container. The optional "gpu" field controls which GPUs are exposed:

gpu value Docker flag Use case
(not set) --gpus all All available GPUs (default)
1 --gpus 1 Limit to 1 GPU
"device=0,2" --gpus "device=0,2" Specific GPU devices

Example driver metadata:

{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/go2-ros2-driver:humble",
      "prefer_gpu": true,
      "gpu": "all"
    }
  }
}

If the NVIDIA runtime is available but not configured as the default in daemon.json, Edge Core logs an informational message with setup instructions instead of silently skipping GPU passthrough.

Jetson detection

Edge Core auto-detects NVIDIA Jetson hardware via /etc/nv_tegra_release, and reads the L4T major release (R36 = JetPack 6, R38 and up = JetPack 7) from its first line. The release gates the ML worker image only — the jetson- driver images below build from the same ROS base as their plain counterparts and differ only in a resource profile (smaller pointcloud budget, software H.264, capped publish rates), so they link no CUDA and carry no L4T floor. When running on a Jetson:

  • The platform key linux-aarch64-jetson is added to the driver resolution order, allowing asset metadata to specify a Jetson-optimised image.
  • If no linux-aarch64-jetson driver key exists in metadata, Edge Core rewrites the image tag by prepending jetson- (e.g. cyberwaveos/go2-ros2-driver:humble → cyberwaveos/go2-ros2-driver:jetson-humble). If the Jetson-prefixed image is not available, it falls back to the original tag automatically.

Override detection with CYBERWAVE_PLATFORM_VARIANT=jetson for testing.

Multi-camera orchestration

When multiple cameras are connected to the same edge device (each represented as a separate digital twin), Edge Core orchestrates them as follows:

  1. One driver per camera: Each camera twin gets its own cyberwave-driver-{uuid[:8]} container. Child camera twins that are attached to a parent twin share the parent's driver instead.

  2. One shared worker: A single cyberwave-worker-{env[:8]} container receives frames from all cameras. The worker container receives CYBERWAVE_TWIN_UUIDS as a comma-separated list of all linked twins.

  3. Readiness probes: Edge Core waits for all driver containers to reach a running state before starting the worker. If some drivers fail, the worker starts anyway so healthy cameras can be utilized.

  4. Model pre-download: Before the worker starts, Edge Core scans worker scripts and pre-downloads all referenced ML models.

  5. Driver health monitoring: when a driver container that was running is found stopped or removed, reconcile_driver_health_for_worker raises a driver_health warning on its twin, whether or not the worker is running.

  6. One device per camera: cyberwave edge install binds each camera twin to its own camera and records the binding under twin_to_device in the camera config. For a USB camera the CLI pins a /dev/v4l/by-id link, or a /dev/v4l/by-path link when two identical units are attached and a by-id name could match either; the value can also be an index (a camera that cannot be pinned), serial:<serial> (a depth camera that the driver resolves by serial) or a stream URL. On Linux, Edge Core passes it to the driver as CYBERWAVE_METADATA_VIDEO_DEVICE (_load_selected_camera_device: an index becomes /dev/video<N>, a link or URL goes through verbatim, a serial: value injects nothing), and a twin missing from the map gets the global selected_device instead. That fallback is why the installer records every pinned twin, even one whose camera the operator chose not to rebind, and every network camera with its stream URL. The injected /dev node is also watched, as described below.

    A by-id link resolves inside the container only if the container can see /dev/v4l. _ensure_linux_camera_docker_params therefore bind-mounts the host's /dev/v4l read-only into every image whose name contains camera-driver (or camera_driver), and into any other image whose final CYBERWAVE_METADATA_VIDEO_DEVICE, injected or set in its own params, lies under /dev/v4l/, unless the params already mount it.

    Three precedence rules decide what the driver actually receives. _read_cameras_config prefers edge.json: a non-empty metadata.cameras there wins over the legacy cameras.json for every key and every twin it lists, and a twin listed only in cameras.json's twin_to_device is merged in, so it keeps that entry instead of falling back to selected_device. An explicit -e CYBERWAVE_METADATA_VIDEO_DEVICE=... in the driver's params wins over the mapping, and Edge Core then injects nothing. And the ALSA audio images, those whose name contains generic-microphone, microphone-driver, generic-speaker or speaker-driver, never receive the variable at all.

  7. Network cameras: a camera on the Ethernet port (a FLIR AX8, an IP camera) has no /dev node. cyberwave edge install finds it on the wire, identifies it by MAC and writes its stream URL into the twin's metadata (video_device, plus mac_address) and into twin_to_device, so Edge Core hands it to the camera driver like any other source; a URL has no node, so it is not watched for presence. When a driver param still names a source, that param wins as above, which is why the installer removes a placeholder or superseded param from the twin's drivers once it binds the device. How the device is found, matched and pinned, and how to describe a new kind of IP device in asset metadata, is in the CLI's wired network devices guide.

Use cyberwave edge status to see all driver and worker containers with their twin mappings.

Camera pins: drift and missing devices

Two reconcilers in the runtime loop (every ~15 s) keep camera drivers honest about the device they were given.

Pin drift. On Linux, reconcile_camera_config_drift notices a camera config change by a new modification time on edge.json or cameras.json. It then compares the CYBERWAVE_METADATA_VIDEO_DEVICE each running driver container was started with against the device its twin resolves to now: the twin's own twin_to_device entry, or the global selected_device when the twin has no entry or its entry resolves to no device (a serial: pin, for example). A container started without the variable is skipped, and so is a twin for which neither names a device. When the two differ, for example after cyberwave edge install --reconfigure-camera, Edge Core restarts the whole edge on a background thread through _perform_edge_core_restart: every driver, the worker and the Zenoh router are stopped and started again, not just the drifted driver. Microphone and speaker containers are never started with CYBERWAVE_METADATA_VIDEO_DEVICE, so they cannot trigger that restart, but it recreates them along with everything else.

Missing devices. reconcile_sensor_device_presence watches each twin that has a twin_to_device entry and a driver container started by this Edge Core, and it checks the device the launcher injects for that twin: the entry is resolved by _load_selected_camera_device, exactly as at launch, so an index becomes /dev/video<N>, a stable link is kept and a serial: pin resolves to nothing. Only a result under /dev/ is checked; serial pins and stream URLs have no node to look for. A node missing on two consecutive checks (about 30 s) raises one sensor_device_missing alert on the twin (error, source_type: "edge", metadata.device naming the path), and two consecutive checks that find it again resolve the alert, so a single glitch neither raises nor resolves one. The alert is also resolved as soon as the twin is no longer checked, because its entry was removed, it was re-pinned to a value with no /dev node, or no driver container is mapped to it any more. Camera drivers react to an unplugged device by reconnecting quietly, so without this check nothing told the operator why a camera had gone dark. Microphones are left out on purpose, because the CLI stores their pin in the twin's audio_device rather than in twin_to_device, and the microphone driver already raises its own sensor-failure alert while its pinned unit is absent. The "already alerted" state lives in memory, so after an Edge Core restart the first check that finds a twin's device present also resolves any alert left from before the restart, once per twin. A resolve that fails, for example while the backend is unreachable, is retried until it succeeds; an alert that fails to send is not retried.

Multi-container drivers

Some robots require multiple cooperating containers (e.g. a driver, bridge nodes, Nav2, SLAM, and elevation mapping). Edge Core supports this via an optional services array in the driver metadata. When present, Edge Core launches one container per service instead of a single driver container.

Metadata schema

{
  "drivers": {
    "linux-aarch64-jetson": {
      "services": [
        {
          "image": "cyberwaveos/go2-ros2-driver:jetson-humble",
          "name": "driver",
          "command": ["ros2", "launch", "cyberwave_go2_driver", "robot_driver.launch.py"]
        },
        {
          "image": "cyberwaveos/go2-ros2-driver:jetson-humble",
          "name": "bridges",
          "command": ["ros2", "launch", "cyberwave_go2_driver", "robot_bridges.launch.py"]
        },
        {
          "image": "cyberwaveos/ros2-nav2:jetson-humble",
          "name": "nav2"
        },
        {
          "image": "cyberwaveos/ros2-slam:jetson-humble",
          "name": "slam"
        },
        {
          "image": "cyberwaveos/ros2-elevation-mapping:jetson-humble",
          "name": "elevation",
          "prefer_gpu": true
        }
      ],
      "shared_env": {
        "CONFIG_PROFILE": "jetson",
        "ROS_DOMAIN_ID": "0",
        "CYBERWAVE_MAP_DIR": "/data"
      },
      "shared_params": ["--network", "host", "-v", "/data:/data"]
    },
    "default": {
      "docker_image": "cyberwaveos/go2-ros2-driver",
      "prefer_gpu": true
    }
  }
}

How it works

  • services present → multi-container mode (one container per service entry).
  • docker_image present, no services → single-container mode (existing behavior, unchanged).
  • The default fallback key works as before for platforms that don't match a specific key.

Per-service fields

Field Required Description
image Yes Docker image reference
name Yes Service name (used in container naming)
command No Override the container entrypoint command
env No Per-service environment variables
params No Per-service Docker params
prefer_gpu No Enable GPU passthrough for this service
gpu No GPU device selector (default: all)

Shared configuration

  • shared_env: Environment variables applied to every service. Per-service env overrides shared values.

  • shared_params: Docker params applied to every service (e.g. --network host, volume mounts).

  • service_image_overrides: for a driver whose image declares its own service graph (no inline services), maps service names to image references. Only those services' images change; the declared graph still supplies commands, env, networks, volumes and dependencies. Use it to try one service's build (a mapper, say) on a robot without restating the whole topology. A name the graph does not declare, or a value that is not a non-empty string, stops the driver from starting.

    "service_image_overrides": {
      "slam": "cyberwaveos/ros2-slam:jetson-humble-<tag>"
    }
    

Environment layering

  1. Edge Core base env (CYBERWAVE_API_KEY, MQTT, Zenoh, etc.) — existing, unchanged
  2. shared_env from metadata
  3. Per-service env from metadata

Container naming

  • Single-container mode (unchanged): cyberwave-driver-{twin_uuid[:8]}
  • Multi-container mode: cyberwave-driver-{twin_uuid[:8]}-{service_name}

Backward compatibility

The existing single-image contract is fully preserved. Metadata without a services key follows the original code path with zero changes. All existing tests continue to pass unmodified.

Writing compatible drivers

A Cyberwave driver is a Docker image that interacts with device hardware and the Cyberwave backend. When Edge Core starts a driver container it sets the following environment variables (provided to the container):

  • CYBERWAVE_TWIN_UUID
  • CYBERWAVE_API_KEY
  • CYBERWAVE_TWIN_JSON_FILE (writable file path; set when the twin file exists at launch, see Twin JSON file)
  • CYBERWAVE_CHILD_TWIN_UUIDS (optional, comma-separated)
  • CYBERWAVE_DATA_BACKEND — data transport backend (zenoh by default)
  • ZENOH_SHARED_MEMORY — true/false; enables zero-copy Zenoh SHM transport
  • ZENOH_CONNECT — (optional) comma-separated Zenoh router endpoint URLs
  • CYBERWAVE_METADATA_VIDEO_DEVICE — (Linux, optional) the camera device this twin is pinned to, from twin_to_device or the global selected_device. It is never set for microphone or speaker images, and it is left alone when the driver's own params already set it. When it names a /dev/v4l/ link, the container also gets the host's /dev/v4l mounted read-only so the link resolves, whatever the image (see Multi-camera orchestration).

Any other -e CYBERWAVE_METADATA_<KEY>=... in a driver's params reaches the container as written and, by the convention the Cyberwave camera and microphone drivers follow, outranks the twin's metadata.<key>. That makes such a param the strongest way to fix a source, and also the easiest way to leave a stale one behind: a binding written to the twin's metadata is ignored while the param is there. Hardware found on the network is bound through the twin's metadata (the keys its asset's network_discovery.bind names, plus mac_address), so a driver for a new kind of IP device should read those keys from the twin JSON file below.

CYBERWAVE_CHILD_TWIN_UUIDS is present when child camera twins are attached to the driver twin; drivers can use this to coordinate cameras without additional prompts.

Zenoh data bus

Edge Core automatically injects Zenoh transport configuration into every driver container so that drivers using cw.data.publish() work without any extra configuration. The data-bus variables are:

Variable Default Description
CYBERWAVE_DATA_BACKEND zenoh Data transport: zenoh or filesystem
ZENOH_SHARED_MEMORY false Opt-in zero-copy shared-memory transport. Requires --ipc=host between containers; leave disabled unless your runtime is configured for it.
ZENOH_CONNECT (empty) Router endpoints, e.g. tcp/10.0.0.1:7447

All variables can be overridden per-driver with -e KEY=VALUE in driver params.

Peer-to-peer mode (default): when ZENOH_CONNECT is empty, Zenoh uses multicast discovery. On Linux with --network host (the default), all driver containers on the same machine discover each other automatically.

Router mode (optional): set ZENOH_ROUTER_ENABLED=true to have Edge Core start an eclipse/zenoh:1.9.0 router container before the driver containers. This is required for MQTT bridge or multi-hop topologies.

Environment variables for Zenoh infrastructure:

Variable Default Description
ZENOH_ROUTER_ENABLED false Start a Zenoh router container before drivers
ZENOH_ROUTER_IMAGE eclipse/zenoh:1.9.0 Docker image for the router
ZENOH_ROUTER_PORT 7447 Host port for the router
ZENOH_SHARED_MEMORY false Opt-in shared-memory transport. Requires all Cyberwave containers to share an IPC namespace (--ipc=host); leave disabled unless validated end-to-end.

Host telemetry

Edge Core reports the host's resource pressure over two transports. The bootstrap publisher folds the gauges into every cyberwave/twin/{uuid}/edge_health heartbeat it sends (~5 s), but Edge Core stops that publisher once driver startup has brought up at least one driver, because drivers publish their own health on the same topics from then on. After that the gauges travel only on the REST host-facts keepalive, which uploads them every ~30 s together with the static hardware facts. In practice, then, MQTT carries host metrics while the edge boots (and beyond that only on an edge whose startup brought up no driver), and REST carries them the rest of the time.

Dynamic (MQTT edge_health payload, every ~5 s while the bootstrap publisher runs):

Field Source Notes
host_memory_percent /proc/meminfo Absent on non-Linux hosts. From the monitor's 30 s snapshot.
host_memory_available_mb /proc/meminfo Free + buffers/cache. From the monitor's 30 s snapshot.
cpu_temp_c /sys/class/thermal/thermal_zone* Hottest CPU-typed zone. From the monitor's 30 s snapshot.
host_cpu_percent /proc/stat Busy share of all cores between the monitor's last two snapshots. Absent on non-Linux hosts and before the second snapshot.
host_disk_percent shutil.disk_usage Usage of the filesystem holding Docker's data root. From the monitor's 30 s snapshot.
consecutive_critical SystemResourceMonitor Consecutive 30 s readings over a critical threshold (memory 92 %, CPU 82 °C). Resets on the first reading below both.
power_mw sysfs power rails Read fresh on every heartbeat; see Resource monitoring.
battery_wh CYBERWAVE_BATTERY_WH Only when set.
watchdog_layers ProcessWatchdog.active_layers() E.g. ["systemd"] or ["systemd", "hardware"]. Also on host_facts, which is where the dashboard reads it — see below. Still published here so a dashboard can keep showing layers for an edge whose binary predates that upload.

Only the bootstrap publisher on the edge host injects these fields. Driver containers see their container's /proc, not the host's, so they intentionally omit the host block and publish only stream-level health.

Per-stream config:

Each entry under streams[id] may carry an optional stream_config block — a discriminated union keyed by kind (camera / audio / lidar / imu) describing the runtime-negotiated config for that stream. Drivers attach a block once at startup via EdgeHealthCheck.register_stream_config(stream_id, config) in the Python SDK; the publisher merges it into every heartbeat thereafter. The legacy top-level camera_config slot stays populated for one deprecation release for out-of-tree consumers and mirrors the lexicographically-first camera-kind stream_config. New consumers must read streams[id].stream_config instead — the legacy slot only ever describes one camera and is silent for audio / lidar / IMU streams.

reconcile_camera_config_drift works from file modification times and docker inspect output and never reads an edge_health payload, so changes to this schema cannot affect it (see Camera pins: drift and missing devices).

Static (REST POST /api/v1/edges/discover, every ~30 s):

Edge Core uploads a host_facts object — total RAM, CPU model and core count, kernel, thermal source, /dev/watchdog availability and the active watchdog layers, non-loopback network interfaces, a primary MAC address, the board serial number, and the current CPU-temperature / memory / power gauges (plus battery_wh when set) — into Edge.metadata['host_facts']. Those gauges duplicate the heartbeat fields above on purpose: the bootstrap publisher stops at the first driver start, and a host row that lost a reading there looked like a fault rather than a transport change. The first POST is performed by a background daemon thread as soon as it starts, then the same thread re-reads and re-POSTs every ~30 s (a fresh read each tick, not a cached payload — the interface list and the gauges below depend on that). Running the bootstrap POST off the main thread is deliberate: under systemd Type=notify, the service signals READY=1 from the main thread, and we don't want a slow or unreachable backend to push past TimeoutStartSec (which would cause systemctl restart cyberwave-edge-core to fail). The call is idempotent (it upserts Edge.metadata['host_facts'] and bumps Edge.last_seen_at), so the keepalive cost is one tiny round-trip per edge regardless of how many bound twins the edge has — or whether it has any at all.

network_interfaces — each interface's name, IPv4 address, MAC and up/down state — exists so an edge that reconnects to a different network (e.g. a different WiFi AP) can still be found and SSH'd into without scanning the LAN. Because it's the one field here that legitimately changes over the device's lifetime, Edge Core additionally publishes a one-line driver_log message ("Network interfaces: eth0=192.168.1.42 (up), ...") to every twin bound to the edge whenever the observed interface set changes, so the change shows up in that twin's Logs tab with a timestamp — not just as a silently-overwritten snapshot.

Three properties of that fan-out are load-bearing:

  • Dedup is per twin, not per process. The set of bound twins grows over an edge's lifetime, so "already published" has to be a per-twin fact — a twin bound after the first publish has never seen the line and still gets one baseline message. Signatures live in memory only, so a restart re-publishes one baseline line per twin.
  • A twin's signature is recorded only after its message reached a connected broker. The publish path swallows failures, so recording up-front would mean a broker that happened to be unreachable on the tick where the IP changed loses that line permanently. Nothing is recorded on failure, so every later tick retries until it lands.
  • The fan-out runs on its own daemon thread, never on the keepalive thread. An MQTT connect against an unreachable broker can burn ~50 s (3 attempts × CONNACK timeout + backoff), which inline would stall the next /discover POST and eat the margin on Edge.last_seen_at — a liveness signal whose whole point is to be independent of MQTT. Only one attempt is ever in flight; a tick that finds one running skips and retries later.

Interfaces with no IPv4 address bound are excluded from both the message and the change signature. They're nothing anyone can SSH to, and they churn (docker0/veth* as driver containers cycle, usb0 as cables move), which would otherwise re-fire the fan-out on changes that tell an operator nothing.

The Edges tab reaches the same goal by a different route: it renders only the interface named by primary_interface_name and keeps the full inventory on hover, so an unplugged NIC's MAC (what you pin a DHCP reservation to) and a VPN interface (how you reach a remote edge) stay one hover away without either crowding out the routable address. The exclusion here is specifically about log-line churn.

That Edge.last_seen_at field is what powers the three-state dashboard pill:

State Trigger Meaning
Online Edge.last_heartbeat fresh (≤60 s) A bound twin is publishing MQTT edge_health.
Standby Edge.last_seen_at fresh (≤90 s) but no recent MQTT Edge Core daemon is alive and posting REST keepalives; no twin is doing work.
Offline Neither signal within window Process is down, or partitioned from both transports.

The 30 s cadence is deliberately tighter than the 90 s Standby window so a single missed POST (slow DNS, transient TLS handshake) doesn't drop the row to Offline — three missed POSTs do. Failure of any individual upload logs at debug and falls through to the next tick.

Coverage per platform:

Field Linux macOS Notes
platform, kernel yes yes From platform.platform() / platform.release().
memory_total_mb /proc/meminfo sysctl hw.memsize Rounded to one decimal on both platforms.
cpu_model /proc/cpuinfo sysctl machdep.cpu.brand_string E.g. "Apple M2 Pro".
cpu_count /proc/cpuinfo (processor: records) sysctl hw.logicalcpu → hw.ncpu Logical CPU count on both platforms (after SMT expansion).
thermal_source first CPU thermal zone omitted Tracks the path the live publisher reads from. Darwin has no thermal reader today, so the field stays absent rather than claim a source that isn't sampled.
has_hardware_watchdog /dev/watchdog always false Linux-only kernel capability. Says the device node exists, not that anything is pinging it — prefer watchdog_layers.
watchdog_layers ProcessWatchdog.active_layers(), overlaid by Edge Core same Which layers are actually supervising this process, in protocol order. The one key here the SDK's read_host_facts() does not produce: it probes /dev/watchdog and so cannot see systemd. Omitted, never [], when no layer is enabled, so "unprotected" and "edge predating this upload" stay distinguishable — the dashboard falls back to has_hardware_watchdog only for the second. Note that "hardware" is absent from the bootstrap POST even on a board that has the device: HardwareWatchdog.enabled only flips inside open(), which main.py calls after run_startup_checks() returns, so the layer appears one keepalive tick later. The dashboard renders that gap — and a permanently failing open() — as watchdog: systemd (+hw idle). Rides REST rather than the heartbeat for the same reason the gauges below do.
network_interfaces /sys/class/net/* (name, address, operstate; IPv4 via SIOCGIFADDR) ifconfig -a (name, ether, status:, inet) The full inventory, deliberately unfiltered beyond loopback — a VPN interface (wg0, tailscale0) is often how you reach a remote edge, so this is not the place to decide what's "reachable". ipv4_address is null when the interface has no bound address (e.g. down); mac_address stays populated, since it's what a DHCP reservation keys on. On Linux, expect it to be long: an edge running driver containers reports 20+ entries (docker0, a br-* per compose network, a veth* per container, kernel tunnel pseudo-devices). Consumers should render primary_interface_name and keep the rest behind a disclosure. macOS narrows ~30 interfaces to the real NICs using three signals read out of the ifconfig text: a name prefix for the families Apple names consistently (utun*, awdl*, bridge*, anpi*, …); media: none, which means no physical layer at all and catches the internal anpi peers that are named en6–en9; and membership of a bridge*, which catches the Thunderbolt ports en2–en5 (the bridge stanza enumerates its own members, so their names are not assumed). An unplugged real NIC reports media: autoselect (none) — a media layer with no link — and is deliberately kept, since its MAC is what a DHCP reservation keys on. Measured on an M-series Mac: 32 interfaces → 3 (en0, en1, and one unplugged port), payload 587 B. Link state comes from status: rather than the RUNNING flag, which Apple's internal NICs assert permanently.
primary_interface_name interface holding the default route interface holding the default route Name of the one interface the host actually routes out of, resolved by asking the kernel for the source address it would use (a UDP connect() to TEST-NET-1 — no packet is transmitted), then matching that address back to the enumerated set. Structural rather than heuristic: a bridge or veth peer cannot hold the default route, so no name blocklist is needed. Falls back to a hardware-backed interface (/sys/class/net/<if>/device exists) that is up when the host has no route at all. Omitted when the MAC came from the uuid.getnode() last resort.
primary_mac_address MAC of primary_interface_name → uuid.getnode() MAC of primary_interface_name → uuid.getnode() The MAC — one value, resolved the same way on Linux, macOS, Jetson and Raspberry Pi, and the one to put in a DHCP reservation. Consumers need no selection logic of their own; picking client-side from network_interfaces by enumeration order lands on docker0 (sysfs enumeration is alphabetical and container bridges are up). Falls back to uuid.getnode() when nothing was enumerated — not tied to a reachable interface, but the only MAC available there; primary_interface_name is then omitted so the two provenances stay distinguishable. Omitted entirely when getnode() fabricated a random node ID (RFC 4122 flags this with the multicast bit), since a MAC-shaped string matching no NIC is worse than nothing.
device_serial Serial in /proc/cpuinfo (Raspberry Pi) → /sys/firmware/devicetree/base/serial-number (Jetson) omitted Board serial for asset tracking / RMA. Device-tree values are NUL-terminated and stripped. All-zero placeholder serials (0000000000000000) are treated as absent.
cpu_temp_c hottest CPU thermal zone omitted Live gauge, not a fact — see below. Deliberately the hottest zone, while thermal_source names the first; the two disagree by design.
memory_used_percent, memory_available_mb /proc/meminfo omitted Live gauges — see below. sysctl gives Darwin the total but not the free/available split, so both stay absent there.
sdk_version, edge_core_version in-process __version__ → importlib.metadata in-process __version__ → importlib.metadata Version of the SDK and Edge Core actually loaded by the running daemon. The in-process lookup preserves the BUILD_VERSION stamp injected by CI for .deb builds and pre-release wheels, and survives PyInstaller binaries that ship without .dist-info, so the Edges tab agrees with cyberwave-edge-core --version. CLI version is intentionally not part of host_facts — the CLI ships as a separate PyInstaller binary, so Edge Core cannot observe it from its own Python process; cyberwave --version remains the source of truth for the CLI an operator is running locally.

Why two gauges ride a facts payload

cpu_temp_c and the memory_* gauges are the same readings the MQTT edge_health heartbeat publishes every ~5 s, duplicated onto the ~30 s REST keepalive. The duplication is not redundancy for its own sake — without it those readings have no route off a working edge at all:

  • The bootstrap edge_health publisher is the only thing that samples them, and Edge Core stops it as soon as the first driver starts (drivers own the twin's health topic from then on).
  • A containerised driver sees its own /proc and has no access to the host's thermal sysfs, so it cannot take over the sampling.

The net effect before this existed: memory and temperature appeared while an edge sat idle and vanished the moment it started doing work — exactly backwards from when an operator wants them. Consumers that may see both copies should prefer whichever is fresher, which is the heartbeat's only while heartbeats are still arriving — a consumer holding the last-received payload in memory keeps those numbers indefinitely, so once the heartbeat stops they become a frozen snapshot and this copy is the live one. The dashboard treats a stale heartbeat as having no gauges at all and falls through to these. Treat a fallback reading as up to 30 s stale at the source, plus whatever polling interval the consumer adds on top.

Local pages and the edge local portal (opt-in)

A driver that serves a page of its own on the edge's network (a download page for a ground-station browser, a camera preview, a status page) declares it in its cw-driver.yml:

local_http:                 # shorthand for drivers.default.local_http
  - name: plans             # slug, unique per driver
    port: 8765              # the port the driver itself listens on
    path: /                 # absolute path; no query string or fragment
    service: gateway        # optional: which service of a multi-container driver serves it
    audience: lan           # lan (default) is listed; host is declared but never listed
    scheme: http            # http (default) or https
    description: Plans for the ground station

The backend validates the block and stores it on the driver profile (drivers.<profile>.local_http); a twin can override it through its own metadata.drivers, with the same precedence Edge Core uses to pick the image. The driver still opens its own port: the declaration only makes it discoverable.

With the portal switched on, Edge Core serves a small read-only index of those pages:

Variable Default Meaning
CYBERWAVE_EDGE_LOCAL_PORTAL off on / true / 1 starts the portal. Anything else leaves Edge Core with no HTTP surface.
CYBERWAVE_EDGE_LOCAL_PORTAL_BIND 0.0.0.0:8700 host:port, [v6]:port, :port or port. Use 127.0.0.1:8700 to keep it on the host.
CYBERWAVE_EDGE_LOCAL_PORTAL_HOST address the client reached Host name or IP printed in links, e.g. a DNS name your LAN resolves.
CYBERWAVE_EDGE_LOCAL_PORTAL_REFRESH_SECONDS 10 How often the listing is rebuilt from docker ps and the twin files.

Set them in the Edge Core service environment or under envs in credentials.json, then restart Edge Core.

  • GET / lists, per running driver, its twin name, registry id, health (from the container healthcheck) and a link to http://<edge-ip>:<port><path> for each declared page. A page whose service container is not running is shown as stopped, without a link.
  • GET /portal.json is the same listing, machine-readable (schema: cyberwave.edge.local_portal.v1).
  • GET /healthz reports the portal's own liveness.

What it does not do, by design:

  • No proxying. Links point straight at the driver's port, so the portal adds an index and nothing else to the edge's attack surface. Protecting a page (if it needs it) is the driver's job.
  • No secrets. The output is built from a fixed set of fields. Tokens, shared_env, image names and the rest of the twin metadata never reach it, and declared paths cannot carry a query string. Entries are re-validated on the edge, because twin metadata is editable after the catalog checked it.
  • No mDNS. Edge Core has no mDNS dependency and this does not add one; reach the portal by IP or by a name your network already resolves.
  • Never blocks the runtime loop. Docker and the twin files are read by a background refresher; requests are answered from that snapshot. A bad bind value or a busy port is logged and Edge Core runs on without the portal.

The links use the address the client connected to (or CYBERWAVE_EDGE_LOCAL_PORTAL_HOST), never the request's Host header.

Driver failure handling

Drivers should exit with a non-zero code when they cannot access required hardware at startup (for example, missing /dev/video* or disconnected peripherals). This allows Edge Core to detect startup failures and trigger restart logic. Not every driver does: the camera driver keeps running and reconnects when its device is unplugged, which is why Edge Core watches camera pins itself (sensor_device_missing below).

Edge Core raises these twin alerts, all with source_type: "edge":

Alert type Severity Raised when Cleared by Edge Core
driver_starting info A driver start attempt begins; carries pull and startup progress (see Remote restart). Resolved once the container runs. A failed attempt is annotated with its phase, escalated to critical and resolved.
worker_starting info The same for the ML worker, held until the worker reports ready (see Worker readiness). As above.
driver_start_failure error The twin has no asset, the driver image cannot be pulled, or the container does not reach a running state. A launch refused only by its restart floor raises nothing. Resolved by the next successful start of that twin's driver.
driver_restart_loop error The driver restarted more than CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD times within CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS. The container is switched to --restart=no and stopped. No.
driver_health warning A driver container that was running is found stopped or removed. No.
sensor_device_missing error The /dev node a managed camera twin is pinned to is missing on two consecutive checks (see Camera pins: drift and missing devices). Resolved when two consecutive checks find it again, or when the twin is no longer checked.
driver_restart_needed warning A configuration key in the metadata of a twin whose driver is not a sensor driver, or declares a restart floor, changed (see Bidirectional twin sync). Edge Core restarts nothing. No.
driver_metadata_restart_suppressed warning A sensor driver has already been restarted three times in 30 minutes for metadata changes; the change is left for a manual restart. No.
edge_cpu_saturation, edge_memory_saturation, edge_disk_saturation warning Raised on every managed twin when CPU, memory or disk usage is at or above CYBERWAVE_RESOURCE_SATURATION_PERCENT on two consecutive 30 s snapshots (see Resource monitoring). Resolved after two consecutive snapshots five points below the threshold.
model_download_failure error A required ML model cannot be downloaded. No.
worker_start_failure error The ML worker container cannot be started, or its startup probe gives up (see Worker startup probe). Resolved by the next successful worker start.
workflow_sync_failure error Downloading a twin's workflow workers from the backend failed. No.
macos_camera_not_configured, macos_microphone_not_configured, macos_speaker_not_configured warning On macOS, a camera, microphone or speaker twin has no host stream or bridge URL configured. No.

When a model fails to download, the worker container is not started: a workflow that depends on the model is held off rather than allowed to run with empty inference results. Models that the worker container can resolve on its own (an Ultralytics auto-download on a frozen PyInstaller binary, for example) are deferred via _RuntimeManagedDeferred and never trip the alert.

docker_create_timeout is a failure phase, not an alert type. When docker create times out and the container has still not appeared 10 s later (see When docker create times out), the driver_starting alert is closed with phase: "docker_create_timeout" and the twin gets a driver_start_failure. Nor does Edge Core create the edge_core_restart alert: the backend creates it when a restart is requested through POST /api/v1/edges/{uuid}/restart-core and ships its UUID in the MQTT command, and Edge Core only moves its metadata.phase from in_progress to completed or failed (EdgeCoreRestartAlertContext in utils.py).

Optional environment variables to tune restart behavior:

  • CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD (default: 4)
  • CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS (default: 60)
  • CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS (default: 30)
  • CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS (default: 300; 0 keeps removed drivers down)
  • CYBERWAVE_DRIVER_TROUBLESHOOTING_URL (default: https://docs.cyberwave.com)

Periodic Docker cleanup

Edge Core prunes stopped cyberwave-* containers every 30 minutes, and unused images and anonymous volumes every 3 hours, from the runtime loop (_run_periodic_docker_cleanup). When Docker's data root is on an SD card the pruning is deferred to spare the flash, but only until that filesystem is 80 % full; CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP=1 turns it off altogether. The intervals, the threshold and the reasoning are under Disk usage: container logs and cleanup.

Driver revival and orphan containers

When a managed driver exits cleanly (Docker's --restart unless-stopped policy does not auto-revive clean exits), Edge Core's revival reconciler re-runs driver startup for that container so it is recreated. Revival is restricted to driver containers this Edge Core process launched. A service declared restart: "no", or one the flap detector stopped, is left down, unless its relaunch was refused only on its restart floor. If docker ps fails or times out, the health snapshot from the previous cycle is kept and nothing is classified, alerted or revived on the strength of the failed listing.

A managed container that has been removed altogether, by docker rm or by the periodic prune, is revived too, but only after a grace period: 300 s by default, set with CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS, where 0 disables it. The grace leaves an operator time for a deliberate takedown, an image swap or a debugging session before Edge Core brings the driver back. _removed_containers_due notes when each container was first seen removed and hands the overdue ones to the same revival pass as clean exits, so the orphan filter, the flap detector's opt-out and the 60 s revival debounce all apply. A removed container cannot be inspected, so Edge Core records the restart policy it launched each container with, and marks the ones the flap detector stopped as no: a removed restart: "no" or flap-stopped driver is not revived, unless its relaunch was refused on its restart floor. If a revival pass finds no linked twin that still declares a removed container, the container is not forgotten at once: that takes three passes in a row without a declaring twin (_REMOVED_FORGET_AFTER_MISSES), each at least a minute apart, so a twin list that comes back short while the backend is flaky does not cost a driver, and a twin that is really gone costs three passes rather than one a minute. Containers removed by an edge restart are dropped from Edge Core's bookkeeping and never come back this way; the restart relaunches what it still needs.

Stopped cyberwave-driver-* containers that this process did not start, such as those of twins unlinked before Edge Core last started, are treated as orphans and ignored by revival. The periodic prune removes them with every other stopped cyberwave-* container, except those that carry a restart-floor label, or when periodic cleanup is disabled. Without this guard, an orphan would send a revival pass to the backend every minute for a container that no linked twin declares.

Twin JSON file

CYBERWAVE_TWIN_JSON_FILE points the driver at /app/<twin_uuid>.json, a JSON copy of its twin: the twin instance as the API returns it (TwinSchema, including metadata), with the catalog asset (AssetSchema) under asset. Edge Core sets the variable only when the host file exists at launch.

On the host the file is <config_dir>/<twin_uuid>.json, and the container sees it twice: as a writable single-file bind mount at /app/<twin_uuid>.json, and inside the read-only mount of the whole config directory at /app/.cyberwave/<twin_uuid>.json. The difference matters because a single-file bind mount follows the file's inode, not its path. That is why write_or_update_twin_json_file, which runs before every driver launch, deep-merges the backend twin into an existing file and rewrites it in place rather than replacing it: the inode a running container holds stays the one the host sees.

Drivers read the file when they start. The Cyberwave camera driver's entrypoint.sh, for example, exports the twin's metadata as CYBERWAVE_METADATA_* variables once, before the driver process runs, so a change to the file while the driver runs does not reach the driver until it is started again.

Bidirectional twin sync

reconcile_twin_json_file_sync runs on every runtime tick (~15 s) and keeps the host files and the backend in step in both directions.

The push leg looks for files whose SHA-256 changed since the previous tick and sends each one with PUT /api/v1/twins/{uuid}. The payload is every field of _TWIN_UPDATE_ALLOWED_FIELDS present in the file: the whole metadata object along with the name, pose, attachment and calibration fields, but never asset or environment assignments. This is how a driver's own edits to its twin file reach the backend. A file seen for the first time only has its checksum recorded, and the writes Edge Core makes itself, at launch and on a pull, record the new checksum too, so they are never pushed back.

The pull leg covers every file the push leg did not send in that tick. _pull_twin_json_file_from_backend fetches the twin with client.twins.get_raw and, for each field in _TWIN_PULL_ALLOWED_FIELDS (only metadata), compares the backend value with the local one as a whole; if they differ at all, the backend value replaces the local one outright, with no key-by-key merge. A push that failed therefore does not protect a local edit, because the same tick's pull can overwrite it with the backend's metadata. For the same reason a field the edge legitimately writes must never be added to _TWIN_PULL_ALLOWED_FIELDS.

Like the launch path, the pull rewrites the host file in place, so the inode a running container holds stays the one the host sees: /app/<twin_uuid>.json shows the pulled content at once, and what the driver writes there still reaches the push leg. The new content is first written to a temporary file, so a full disk fails before the original is touched, and the original is then overwritten without being emptied first, a shorter document padded with spaces until the old tail is cut, so a reader always finds valid JSON. A host file that cannot be read is rebuilt from the last copy Edge Core wrote or read intact, with the backend metadata applied; with no such copy in memory it is left, with one warning, for the next launch to rewrite. The pull reports which top-level metadata keys changed, and reconcile_twin_json_file_sync returns them per twin.

Drivers read the file only when they start, so a metadata edit made in the UI, such as a new microphone gain, reaches a driver only through a restart. reconcile_driver_metadata_restarts, which runs right after the sync, makes that restart for sensor drivers: the camera, microphone, speaker and webcam drivers, recognised by their image names.

  • Configuration only. Runtime state is left out of the comparison: telemetry (status, power, motor_temps, performance), the backend's navigation keys (active_map_*, finalize_active_map_request, last_finalized_map_*, navigation_lifecycle, localization), the drivers' own reports (speaker_device, audio_routing, robot_dump, and audio_device when it is the webcam's dict rather than a pinned device string) and edge_fingerprint; a change of edge_fingerprint, a link or unlink, never restarts anything. In sensors, entries without parameters are left out too, because a driver registers a missing sensor on its own as a bare id and type, and a list left empty counts as no sensors at all. Every other key is configuration, compared as a digest of the filtered metadata.
  • Debounced. The restart happens once the filtered metadata has been unchanged for at least 10 s, that is on a later tick, so several saves in a row cost one restart.
  • The driver's own writes do not count. The digest written at launch is the baseline, and whatever Edge Core pushes from the file becomes the new one; the writes a driver is known to make through the backend are filtered out of the digest (see above). Each of the twin's containers is judged on its own: one that started after the change, or is not running, has it or reads it when it starts. A change seen during a container's first 90 s waits for those 90 s to pass, so nothing restarts while a driver is starting; if the filtered metadata still differs from what the containers started with, the twin is then restarted. A real edit made right after a restart therefore still reaches the driver, and a startup write Edge Core does not recognise costs at most the restarts the rate limit allows. A container that cannot be inspected holds the decision until a later tick.
  • Relaunch. The twin's containers go through fetch_and_run_twin_drivers(..., only_container_names=..., pull_images=False), the revival path, which re-fetches the twin, rewrites the file and shows driver_starting. A child camera twin restarts the containers of the twin it is attached to. The relaunch uses the image already on disk and pulls only a missing one: a settings change applies settings and does not upgrade the driver. It also runs inside the runtime loop, whose only WATCHDOG=1 comes at the end of each cycle, so a pull of a newly published tag could outlast WatchdogSec=60 and get Edge Core aborted. For the same reason the relaunch gets the loop's watchdog, which the dependency health wait pings. The change counts as applied only when every relaunched container started; otherwise it stays pending and is retried under the rate limit below.
  • Rate limit. At least 120 s between two metadata restarts of a twin and at most three in 30 minutes; beyond that the change is left for a manual restart and the twin gets one driver_metadata_restart_suppressed alert.
  • Never during another restart. The step waits while an Edge Core or camera-drift restart is in progress.

Any other driver, a robot or a drone, and any service with a restart floor, is not restarted for a metadata change: the change raises one driver_restart_needed alert ("Settings changed; restart the driver to apply them") on the twin whose settings changed. Until that alert, or a driver_metadata_restart_suppressed one, reaches the backend, the change stays pending and the alert is retried on later ticks.

Twin metadata

Use the official Cyberwave SDK to interact with the API and MQTT; it abstracts authentication, retries, and handshake logic.

Register a driver by adding its configuration to a twin's metadata (or the catalog twin's metadata if you control the catalog twin). Use the environment view's Advanced editing to edit metadata.

Note: changing a catalog twin's metadata affects all subsequently created digital twins derived from that catalog twin.

Example driver metadata (JSON):

{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/so101-driver",
      "version": "0.0.1",
      "params": [
        "--network",
        "local",
        "--add-host",
        "host.docker.internal:host-gateway"
      ]
    }
  }
}

Platform-specific driver selection

Edge Core can select platform-specific driver entries before falling back to default.

Selection order:

  1. Child-registry-specific entry (existing behavior)
  2. Host platform/machine keys (for example darwin-arm64, darwin, macos, mac)
  3. default

Example:

{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/so101-driver"
    },
    "darwin-arm64": {
      "docker_image": "cyberwaveos/so101-driver:macos",
      "params": ["-e", "CYBERWAVE_SERIAL_BRIDGE_URL=tcp://host.docker.internal:22001"]
    }
  }
}

macOS host-device bridge hook

On macOS, Linux --device mappings in params cannot directly expose host hardware to Linux containers. Edge Core now supports a pre-run native bridge hook:

  • Set CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMAND on the host
  • Edge Core executes it once per --device mapping before docker run
  • Template variables available:
    • {host_device}
    • {container_device}
    • {twin_uuid}
    • {container_name}
    • {config_dir}

Example:

export CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMAND="cyberwave-edge-hw-bridge --device {host_device} --target {container_device} --twin {twin_uuid}"

The command can start native camera/serial forwarding services that expose bridge endpoints to the container (typically via host.docker.internal).

Bridge command stdout can optionally return a resolved source for the mapped device:

  • JSON: {"resolved_device":"rtsp://host.docker.internal:8554/cam0"}
  • or line format: resolved_device=rtsp://host.docker.internal:8554/cam0

When this value differs from /dev/video*, Edge Core can transparently:

  • inject CYBERWAVE_METADATA_VIDEO_DEVICE for the driver
  • inject CYBERWAVE_CHILD_CAMERA_STREAM_URLS with the per-child MJPEG mapping for multi-camera parent drivers such as SO101
  • inject CYBERWAVE_EDGE_VIDEO_DEVICE_MAP (JSON map of Linux device to resolved source)
  • remove Linux-only --device /dev/video* flags before docker run on macOS (default enabled with CYBERWAVE_MACOS_STRIP_VIDEO_DEVICE_PARAMS=true)

This lets Linux-style drivers keep their normal auto-setup logic while receiving a macOS-compatible video source without driver code changes.

For camera twins, Edge Core can also provide default bridge candidates on macOS even when metadata has no explicit --device params (default driver config), so Linux-oriented camera drivers remain compatible with minimal metadata.

To inject environment variables into a driver container, list -e flags inside params: each -e must be a separate element followed by its KEY=value string. Example:

{
  "drivers": {
    "default": {
      "docker_image": "cyberwaveos/go2-native-driver",
      "params": ["-e", "MY_VAR=value", "-e", "ANOTHER_VAR=value2"]
    }
  }
}

Each -e must be its own element in the array, followed by the KEY=value string as the next element. This is equivalent to passing -e MY_VAR=value on the docker run command line.

This is useful for driver-specific configuration that varies per device, such as IP addresses, credentials, or feature flags that cannot be stored in the twin's edge_configs metadata.

Runtime configuration for drivers (metadata["edge_configs"])

Drivers and edge services should treat metadata["edge_configs"] as the source of truth for per-device runtime configuration. Edge identity should be stored at metadata["edge_fingerprint"] (not duplicated inside edge_configs).

Runtime access: The core passes the full twin JSON (including metadata) to every driver via the CYBERWAVE_TWIN_JSON_FILE environment variable. Drivers can read edge_configs from that file at startup to obtain per-device settings — for example, selecting the right camera source or IP address for the current machine. This is the recommended way to pass device-specific configuration to a driver without hardcoding values in the image.

  • Type: object/dictionary
  • Value: binding object (object)

Canonical shape:

{
  "edge_fingerprint": "macbook-pro-a1b2c3d4e5f6",
  "edge_configs": {
    "camera_config": {
      "camera_id": "front",
      "source": "rtsp://user:pass@192.168.1.20/stream",
      "fps": 10,
      "resolution": "VGA",
      "camera_type": "cv2"
    }
  }
}

Field notes:

  • edge_fingerprint: fingerprint of the edge serving this twin (recommended).
  • camera_config: per-device camera/runtime config consumed by drivers.

Avoid storing transient runtime state such as edge_uuid, registered_at, last_sync, last_ip_address, or status_data inside edge_configs.

Backward compatibility:

  • Older records may use a legacy map shape (edge_configs[fingerprint] = {...}).
  • Older records may store camera settings in cameras[0] or as top-level fields.
  • New writers should prefer camera_config under edge_configs.
  • Do not rely on PUT /api/v1/edges/{uuid}/twins/{twin_uuid}/camera-config; it is deprecated. Update twin metadata instead.

Advanced usage

Manual install and troubleshooting

# Install the Buildkite package signing key
curl -fsSL "https://packages.buildkite.com/cyberwave/cyberwave-edge-core/gpgkey" | gpg --dearmor -o /etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg

# Configure the Apt source
echo -e "deb [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main" \
  > /etc/apt/sources.list.d/buildkite-cyberwave-cyberwave-edge-core.list

# Run Edge Core (performs startup checks and starts drivers + worker container)
cyberwave-edge-core

# Show status, credentials and MQTT connectivity (read-only)
cyberwave-edge-core status

# Show version
cyberwave-edge-core --version

# Worker container management (also available via `cyberwave worker …`)
cyberwave-edge-core worker start      # Start the worker container
cyberwave-edge-core worker stop       # Stop the worker container
cyberwave-edge-core worker restart    # Restart the worker container
cyberwave-edge-core worker status     # Show container state, workers, cached models, and health
cyberwave-edge-core worker health     # Show detailed restart history and circuit-breaker state
cyberwave-edge-core worker logs       # Stream worker container logs
cyberwave-edge-core worker logs --no-follow  # Print recent logs without following

Preview builds from dev / staging CI are published as separate Debian packages in the same apt repo: cyberwave-edge-core-dev and cyberwave-edge-core-staging. apt install cyberwave-edge-core only pulls tagged releases; use one of the channel packages explicitly when you want those binaries (the packages conflict because they ship the same /usr/bin/cyberwave-edge-core).

On non-apt platforms, prerelease Python wheels are published to the Buildkite Python registry and consumed automatically by cyberwave edge install --channel dev|staging. Stable pip installs continue to use the public PyPI release.

Environment variables

Run against a different environment/base URL:

export CYBERWAVE_ENVIRONMENT="yourenv"
export CYBERWAVE_BASE_URL="https://yourbaseurl"
cyberwave-edge-core

Control log verbosity (default: INFO). Edge-core forwards any CYBERWAVE_* env var it sees in its own process environment into each worker container at docker run time, so the same knobs work whether they're set in the systemd unit, the CLI install env, or the shell:

# Edge-core + edge drivers (camera/UGV/etc.)
export CYBERWAVE_EDGE_LOG_LEVEL="DEBUG"
# Worker containers only — wins over CYBERWAVE_EDGE_LOG_LEVEL when set,
# so you can keep workers at INFO while a driver is at DEBUG (or vice versa).
export CYBERWAVE_WORKER_LOG_LEVEL="INFO"
cyberwave-edge-core

Or pass env vars to the CLI installer:

sudo CYBERWAVE_ENVIRONMENT="yourenv" CYBERWAVE_BASE_URL="https://yourbaseurl" CYBERWAVE_MQTT_HOST="yourmqtt" cyberwave edge install

Disk usage: container logs and cleanup

Docker's default json-file log driver has no size limit, and every container Edge Core launches runs under --restart unless-stopped. On a device with a small disk — a Raspberry Pi on a 16 GB uSD, say — a chatty driver can fill the card in days. Edge Core therefore caps each container's log at launch rather than relying on the host's /etc/docker/daemon.json.

Variable Default Purpose
CYBERWAVE_DOCKER_LOG_MAX_SIZE 10m Max size per container log file before rotation. Set to off to emit no log flags at all and inherit the host daemon's configuration — use this if you ship logs through a daemon-wide driver such as journald or fluentd.
CYBERWAVE_DOCKER_LOG_MAX_FILE 3 Rotated files kept per container (so 30 MB per container by default).
CYBERWAVE_DRIVER_LOG_COLD_START_TAIL 500 Backlog lines replayed when Edge Core starts following an already-running container's logs. 0 restores the previous unbounded replay.

Per-container flags override daemon.json, so the defaults match what the Pi image already sets daemon-wide — installing Edge Core never raises a cap the host had already chosen. Raise them explicitly if you want more history.

A driver's catalog docker_run_params win, at the granularity they were set: an explicit --log-driver means Edge Core adds nothing at all, while an explicit --log-opt max-size (with no driver) keeps that value and still gets the default max-file bound.

Periodic cleanup prunes stopped Cyberwave containers (every 30 min) plus unused images and anonymous volumes (every 3 h):

Variable Default Purpose
CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP unset Set to 1 to disable all periodic pruning.
CYBERWAVE_SD_CARD_CLEANUP_USAGE_PERCENT 80 When Docker's data root is on an SD card, routine pruning is deferred to limit flash wear — but runs anyway once that filesystem reaches this percentage full.
CYBERWAVE_CONTAINER_PRUNE_INTERVAL_SECONDS 1800 Stopped-container prune interval.
CYBERWAVE_IMAGE_PRUNE_INTERVAL_SECONDS 10800 Unused-image prune interval.
CYBERWAVE_VOLUME_PRUNE_INTERVAL_SECONDS 10800 Anonymous-volume prune interval. Named volumes are never pruned.

Volume pruning requires Docker 23.0 or newer, where docker volume prune removes only anonymous volumes. On older daemons the same command would also delete unused named volumes belonging to other workloads on the host, so Edge Core skips it and prunes containers and images only. The docker-ce packages our installer and Pi image use are well past 23.0; the distro docker.io package is not.

If a device is running out of space, start here:

docker system df -v
sudo ls -lSh /var/lib/docker/containers/*/*-json.log | head
du -sh ~/.cyberwave/models/
journalctl --disk-usage

Local development (from this folder)

You can develop both the Cyberwave CLI and Edge Core from the cyberwave-edge-core directory using a single virtual environment that has the monorepo SDK, CLI, and edge-core installed in editable mode.

One-time setup

From cyberwave-edge-core/:

# Create and activate a venv (e.g. .venv in this folder)
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate

# Install SDK, CLI, and Edge Core in editable mode (order matters: SDK first)
pip install -e ../cyberwave-sdks/cyberwave-python
pip install -e ../cyberwave-clis/cyberwave-python-cli/"[build]"
pip install -e ".[build]"

Generate the SDK REST client (required for editable SDK). The SDK’s cyberwave.rest package is generated from the backend OpenAPI spec and is not committed. If you see ImportError: cannot import name 'DefaultApi' from 'cyberwave.rest':

  1. Start the backend: cd ../cyberwave-backend && docker compose -f local.yml up -d (wait until healthy).
  2. From the repo root, generate the REST client:
    cd cyberwave-sdks && ./python-sdk-gen.sh sdk --host localhost:8000
    
  3. Re-run the pip install -e steps above if you already installed; the editable SDK will then include the generated cyberwave/rest code.

Run CLI and Edge Core

After activating the venv, both commands are on your PATH:

# CLI
cyberwave --help
cyberwave login --email boss@cyberwave.com --password iamnottheboss
cyberwave edge install --help

# Edge Core
cyberwave-edge-core --help
cyberwave-edge-core status
cyberwave-edge-core

Target backend: If you do not set CYBERWAVE_BASE_URL, the CLI and Edge Core use the default production API (https://api.cyberwave.com). To use your local backend instead:

export CYBERWAVE_BASE_URL=http://localhost:8000
export CYBERWAVE_MQTT_HOST=localhost
export CYBERWAVE_ENVIRONMENT=local

Paths from this folder

What Path (from cyberwave-edge-core/)
Repo root ..
Python SDK ../cyberwave-sdks/cyberwave-python
CLI ../cyberwave-clis/cyberwave-python-cli

Edit code in any of those directories; the editable installs pick up changes (no reinstall needed for Python changes).

Testing

CI (.github/workflows/edge-core-test-and-push.yml) runs the whole unit suite, pytest tests/ --ignore=tests/integration, and then the Docker integration tests separately. Run the same thing locally from this folder:

pytest tests/ --ignore=tests/integration -q

The systemd behaviour (readiness, the watchdog, bounded start and stop, upgrades of the Debian package) can only be checked under a real systemd. scripts/test_deb_upgrade_under_systemd.sh boots one in a container, installs the package given by PACKAGE_FILE and checks install, upgrade, hold and update. It then runs the lifecycle test files of the CLI and of Edge Core inside that systemd, with CYBERWAVE_TEST_SYSTEMD_SYSTEM=1 and CYBERWAVE_REQUIRE_SYSTEMD_TESTS=1 so that tests needing systemd fail instead of skipping. The Edge Core checkout is mounted read-only. LIFECYCLE_ONLY=1 skips the Debian checks and needs no package, which is the quick way to check a lifecycle change:

LIFECYCLE_ONLY=1 bash scripts/test_deb_upgrade_under_systemd.sh
PACKAGE_FILE=path/to/package.deb bash scripts/test_deb_upgrade_under_systemd.sh

Contributing

Contributions are welcome. Please open an issue to discuss bugs or feature requests, and submit a pull request when you are ready.

Community and Documentation

Metadata

Release files for cyberwave-edge-core 0.1.39

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cyberwave-edge-core 0.1.39
File Size Uploaded Provenance Actions
cyberwave_edge_core-0.1.39.tar.gz 746.7 kB With provenance Details for cyberwave_edge_core-0.1.39.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cyberwave-edge-core 0.1.39
File Interpreter ABI Platform Actions
cyberwave_edge_core-0.1.39-py3-none-any.whl Python 3 none any Details for cyberwave_edge_core-0.1.39-py3-none-any.whl

Total release size: 1.1 MB

Release files / cyberwave_edge_core-0.1.39.tar.gz

Download URL cyberwave_edge_core-0.1.39.tar.gz
Size 746.7 kB
Tags Source
SHA-256 checksum
How to use checksums
9ca3052926da6d5a7d7a7143c633760644faa5bad3c2c093d47ded797820e3fb
BLAKE2b-256 checksum
How to use checksums
e40ac9ee17ce6d4de0adc4abd8d52ab7563d4977118bdfde6153c66b89113382
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / cyberwave_edge_core-0.1.39-py3-none-any.whl

Download URL cyberwave_edge_core-0.1.39-py3-none-any.whl
Size 390.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f90b59a95b9f3af325ce77aabe947f31f1b987a8882a747edfe9633d4b6bb6df
BLAKE2b-256 checksum
How to use checksums
2c60e26f19c3e662e32249e677c31999d58394f176121ac5938fad208c653139
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for cyberwave-edge-core 0.1.39

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for cyberwave_edge_core-0.1.39.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.39 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page