Cyberwave Edge Core
This module is part of Cyberwave: Making the physical world programmable.
Cyberwave Edge Core acts as the orchestrator of Cyberwave edge drivers.
Quickstart
SSH to the edge device where you want to install Edge Core, then install the Cyberwave CLI and run the installer:
# Install the Cyberwave CLI (one-time setup)
curl -fsSL https://cyberwave.com/install.sh | bash
# Run the edge installer (interactive)
sudo cyberwave edge install
The installer will prompt you to log in with your Cyberwave account, select a workspace and environment, and persist configuration under ~/.cyberwave/. You can override the config directory via the CYBERWAVE_EDGE_CONFIG_DIR environment variable. Legacy installs that used /etc/cyberwave are automatically migrated.
Permissions on the config directory:
credentials.jsonis written with mode0600(owner-only) because it holds your API token.fingerprint.jsonis written with mode0644(world-readable) because the device fingerprint is a hardware identifier, not a secret. This lets user shells read it even when Edge Core runs as root viasystemd.- On
systemddeployments where Edge Core runs as root, the service re-chowns files under~/.cyberwave/on startup so they stay owned by the user whose home directory holds them.
Don't have a Cyberwave account? Get one at cyberwave.com
Config files created
The installer and Edge Core create these files in the config directory:
| File | Description |
|---|---|
credentials.json |
API token and workspace information |
fingerprint.json |
Device fingerprint (generated by Edge Core) |
environment.json |
Selected environment and twin UUIDs |
edge.json |
Cached edge device record from the backend (written on registration, refreshed periodically) |
{twin-uuid}.json |
Cached twin + asset record (one per twin) |
cameras.json |
Deprecated. Legacy camera device config; superseded by edge.json metadata.cameras |
Edge Core requires credentials.json to operate. fingerprint.json is produced by Edge Core; environment.json is written by the CLI during setup. edge.json is written on registration and stores the full edge device record, including metadata.cameras (the device list and the twin-to-device mapping). Every ~5 minutes reconcile_edge_json_file_sync fetches the record again, but it rewrites the file only when metadata.cameras changed, because a rewrite is what the camera drift check reacts to (see Camera pins: drift and missing devices).
How Edge Core works
On startup (service or direct run), Edge Core performs the following steps:
- Validate credentials from
credentials.json. - Connect to the backend MQTT broker and verify connectivity.
- Start a bootstrap health publisher that sends periodic edge health messages while drivers are starting up.
- Register the edge device and record a unique
edge_fingerprint. - Download the selected environment and resolve twins linked to the fingerprint.
- Start drivers for linked twins. Special handling for attached camera child twins:
- If a twin is a camera child (has
attach_to_twin_uuid), Edge Core does not start a separate driver for it. - Camera child UUIDs are passed to the parent driver via
CYBERWAVE_CHILD_TWIN_UUIDS.
- If a twin is a camera child (has
- Stop the bootstrap health publisher once drivers are running (drivers publish their own health messages; keeping both would produce duplicate signals in the UI).
- Pull workflow workers (
wf_*.py) for the twins listed inenvironment.jsonand start the worker container (if any worker files exist in{config_dir}/workers/).
Scope of workflow worker sync
Workflow worker sync is scoped strictly to the twin UUIDs the operator selected at install time and persisted to environment.json under twin_uuids. This is intentionally narrower than the fingerprint-based discovery used for drivers and the bootstrap health publisher: an environment can carry stale metadata.edge_fingerprint entries from previous installs, and we don't want those to pull unrelated wf_*.py files onto this edge. For backward compatibility, installs that predate the twin_uuids field still fall back to fingerprint-based discovery.
During driver startup, Docker image pull progress is mirrored into the edge-core service logs and forwarded through the same MQTT-backed driver log stream used for runtime container logs, so users can follow image download progress remotely.
Remote restart (Edge REST API)
Request a remote restart of Edge Core via the REST API:
POST /api/v1/edges/{uuid}/restart-core
The API will publish an MQTT message to the edge's command topic:
Topic: edges/{edge_uuid}/command
Example payload:
{ "command": "restart_edge_core" }
When Edge Core receives this command it performs a graceful restart consisting of:
- Stopping the worker container (if running).
- Removing cached twin JSON files from the edge config directory.
- Stopping and removing any edge-managed driver containers, then pruning stopped containers.
- Resolving any active
driver_startingalerts on the affected twins so leftovers from the previous run do not stay visible after the restart. - Re-downloading the selected environment and restarting drivers.
- Restarting the worker container (if worker files exist).
The restart deletes the cached twin JSON files without a final push, so an edit a driver made to its twin file after the last runtime tick (see Twin JSON file) is lost; everything else is downloaded again from the backend.
Each driver startup attempt creates a driver_starting twin alert that tracks the in-flight startup (image pull, container launch, post-launch health probe). The alert is automatically resolved once the driver container is observed running; if the attempt fails, it's annotated with the failure phase (e.g. pull_failed / pull_timeout / pull_oserror), escalated to critical severity, and resolved. The same lifecycle applies to the ML worker via the worker_starting alert, which is raised before the image pull and held until the worker container is observed running — so a worker whose image is already on disk still reports that it is starting, rather than coming up silently. The alert is therefore guaranteed to clear when the driver has restarted; longer-lived failure conditions are surfaced through separate driver_start_failure alerts created by the orchestrator.
While docker pull is running, edge-core writes pull progress directly onto the alert's metadata (progress_percent, downloaded_bytes / total_bytes, downloaded_human / total_human, phase) so the workbench renders e.g. Downloading driver image (cyberwaveos/ugv-driver:dev) — 43% instead of a stale "Still pulling" spinner. The byte counts join the message once every layer has declared its size (745 MB of 1.55 GB); until then the image size is not yet known and only the percentage is shown. The same summary feeds the systemd watchdog (systemctl status cyberwave-edge-core) and extends TimeoutStartSec while bytes are still arriving. The parser, fan-out worker, and tests live in cyberwave_edge_core/driver_logs.py and tests/test_docker_pull_progress.py.
Startup is reported as three distinct stages. overall_progress_percent — the number a progress bar should read — spans all three as a single 0–100 that only moves forward (download 0–45, install 45–80, container 80–100), because the card renders one bar and a stage restarting its own count reads as the bar rewinding. stage_progress_percent carries the same startup as a per-stage 0–100, for a stage-aware bar to use; the alert's text names the stage on screen, so a stage that reads 100 % is one you can check against what you are being told. The bands are sized by how long each stage takes rather than by how many steps it has, so install — the stage that outlasts the download — gets a comparable share of the bar.
Downloading is driven by layers fetched rather than by bytes: docker only declares a layer's size when it starts fetching it, so a byte ratio has an unknown denominator for most of the pull and reads 100 % every time the in-flight batch drains. The layer currently downloading counts as its byte fraction, so the number still moves between layers. 100 % is reserved for a pull docker has confirmed finished. Installing — verifying and decompressing, which on a multi-GB image outlasts the download — tracks layer completion (layers_complete / layers_total). Starting covers container create → run → probe.
Local driver builds are never pulled
A driver image whose tag is local, starts with local- or ends with -local is treated as a developer build. When such an image already exists on the host, Edge Core uses it as it is: _use_local_build takes it out of the parallel pull, the startup alert records phase: "pull_skipped", and the log says Using the local build of <image>; local-only tags are not pulled. Before this rule a restart pulled the tag like any other, and because Docker Hub can carry an image with the same tag, the pull quietly replaced the build you were testing. A local-only tag that is not on the host is still pulled, and digests (@sha256:…) never count as local. The worker has its own, broader rule; see Worker image refresh policy.
Which image a twin's driver runs
The docker_image in the twin's metadata.drivers is what runs, whatever tag it carries (humble-dev-sha-<sha>, humble-0.1.3-rc.7, humble-pr-4549-sha-<sha>, a @sha256: digest, a release such as jetson-humble-0.1.23). After docker ps on the edge, the driver's own services show that reference, not a moving channel tag.
A driver image can declare a stack of sibling images in its io.cyberwave.driver.compose.* label (the Go2 declares go2-ros2-driver, ros2-slam and ros2-nav2). Edge Core resolves that graph as follows:
- Own services. A service of the same repository as the twin's image (
plantandbridgesfor the Go2) runs the twin's reference exactly. The label names it<repo>:${CW_CHANNEL_TAG}, which used to resolve to the channel the image was built in, so a twin onhumble-dev-sha-<sha>ranhumble-dev. See_run_the_twin_image_in_its_own_servicesincyberwave_edge_core/startup.py. - Siblings of a pinned graph. CI pins every sibling to an immutable
-sha-tag and marks itx-cyberwave.pinned_from(pin-driver-compose-images,.github/scripts/pin_driver_compose.py). Edge Core runs those tags as written. That is what keeps driver and siblings on one release. - Siblings of an unpinned graph. If the twin's image is a release tag (
<distro>-X.Y.Z, optionallyjetson-led), the${CW_CHANNEL_TAG}siblings are requested at that same tag; otherwise they follow the image'sio.cyberwave.image.channel-taglabel. See_resolve_channel_tagincyberwave_edge_core/driver_selection.py. PR re-tags (humble-pr-…), dev prereleases (humble-0.1.3-dev.5) and moving tags are not release tags and keep following the label. Siblings keep their own version counters, soros2-slam:jetson-humble-0.1.23is generally not a published image: step 4 refuses this case rather than letting those services fail to pull. The simulator reads the same graph throughget_image_declared_servicesbut has no step 4, so a release-tagged twin there fails to pull those siblings instead of being refused. - No proof of one release, no start. A sibling that is still declared by placeholder after step 2 is whatever its channel resolves to, so the stack may mix releases, or name an image that does not exist. Moving tags (
humble-dev) and version-shaped tags that came from step 3 both count: the check looks at how the graph named the sibling, not at the shape of the tag. Edge Core refuses to start that twin: the log hasTwin '<name>' refused: driver image … does not pin its sibling services to a release … : slam=<image>, and adriver_start_failurealert titled Driver stack is not pinned to one release names the services. Only drivers with sibling images are affected, and only a twin that names one build (a release, a-sha-tag, a digest): a twin on a moving tag such ashumbleasks for a channel, its siblings follow the same channel, and it starts as before.
Ways to run such a twin: use a driver release whose images are pinned; pin each named service in the twin's service_image_overrides (a tag or a @sha256: digest; a service pinned there counts as pinned); or set CYBERWAVE_DRIVER_CHANNEL_TAG on the edge to pin the whole graph, driver services included, to one train (an operator override; it disables steps 1 and 4).
service_image_overrides is applied last and always wins. An image without a registry namespace (a bare local build name) is never refused, because there is nothing to tell its siblings from third-party images such as redis.
Tests: tests/test_image_declared_topology.py and tests/test_resolve_twin_driver_specs.py.
When docker create times out
docker create runs with a timeout (CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS, 30 s by default), but the timeout only kills the CLI. A busy Docker daemon usually goes on to finish the create, which used to leave a container in Created that Edge Core had never registered and therefore never retried. Edge Core now waits up to 10 more seconds for the container to appear (_wait_for_container in docker_launch.py). If it does, the launch carries on as normal, with a warning in the log; only if it does not is the attempt failed with docker_create_timeout.
Model Manager (ML model cache)
Edge Core includes a ModelManager that resolves ML model weights into a local cache before starting the worker container. It is designed for both online deployments and air-gapped sites: it prefers fresh weights from Cyberwave when the network is available, falls back to upstream public mirrors, and finally to whatever is already on disk.
Cache location:
| Platform | Default path |
|---|---|
| All | ~/.cyberwave/models/ |
Override with CYBERWAVE_EDGE_CONFIG_DIR.
Cache layout:
<cache_dir>/
├── manifest.json # index of all cached models
├── yolov8n/
│ ├── yolov8n.pt # weight file
│ └── metadata.json # checksum, runtime, source URL, upstream URL
└── background-subtraction/
└── ...
Model requirements discovery: Edge Core scans *.py files in ~/.cyberwave/workers/ for cw.models.load(...) calls to determine which weights to ensure.
~/.cyberwave/models/ and ~/.cyberwave/workers/ are created eagerly on Edge Core startup (even before any worker runs), with ownership matching the invoking user, so operators can drop pre-staged weights into ~/.cyberwave/models/{model_id}/ from a regular shell.
On systemd deployments where Edge Core itself runs as root, every per-model staging directory, downloaded weight, sidecar metadata.json, and the top-level manifest.json is also chowned to the invoking host user at write time — not just at startup. This keeps the cache both readable and writable from inside the worker container (which launches with --user $UID:$GID); without it, cw.models.load crashes with PermissionError: [Errno 13] on iterdir when traversing a root-owned cache_dir/{model_id}/. Writability matters because the SDK's UltralyticsRuntime sandboxes lazy secondary-asset downloads (e.g. the MobileCLIP text encoder for YOLOE / YOLO-World text prompts) into the weight directory. The wiring lives in ModelManager(owner_uid_gid=resolve_config_owner_uid_gid()).
Resolution order
For each required model, ensure_model(model_id) runs the following steps:
-
Reconcile disk. If
cache_dir/{model_id}/already contains a weight file (with or without a sidecar), it is registered in the manifest. A missing sidecar is generated from the on-disk file (with a freshly computed SHA-256) and taggeddownloaded_from: prestaged. This is how an operator pre-stages weights from a USB stick on an air-gapped site. -
Verify cache integrity. If the local file's SHA-256 matches the manifest checksum, the cache is intact.
-
Pre-staged short-circuit. When an intact entry is tagged
downloaded_from: prestaged, Edge Core returns it without ever contacting the catalog. Pre-staged files are operator-curated truth; to force a re-download, evict the model directory. -
Best-effort catalog probe. For non-prestaged intact entries, Edge Core does a short-timeout
GET /api/v1/mlmodels/...to compare checksums.- Catalog unreachable, no checksum, or matching checksum → return the cached file (no download).
- Catalog returns a different checksum → fall through to the download path.
-
Download. Sources are tried in priority order:
- Cyberwave-hosted signed URL from
GET /api/v1/mlmodels/{uuid}/weights— used for checkpoints we have uploaded to our private GCS bucket (e.g. internally trained or mirrored models). Authenticated, served from infrastructure we control. - Upstream weights URL from the catalog entry (
download_url/metadata.upstream_weights_url) — used for community checkpoints we did not mirror. - Runtime-managed download as a final fallback: when neither URL is set but the catalog entry's
edge_runtimeships its own weight resolver (today:ultralyticsagainst its GitHub releases hub), Edge Core invokes it in a subprocess and caches the resulting file asdownloaded_from: runtime_managed. This is what lets.ptcatalog entries (YOLO26, YOLOE-26) work without a mirrored upstream URL. Requires the runtime to be installed on the edge host — see ML runtime extras.
The first source that yields a checksum-verified file wins. The sidecar records
downloaded_from(artifact_url/download_url/runtime_managed/prestaged),source_url(the public URL we fetched, ornullfor artifact and runtime-managed downloads — neither is a re-fetchable URL we want to persist), andupstream_url(provenance). - Cyberwave-hosted signed URL from
-
Fail-soft. If every download attempt fails and the cached file is intact, Edge Core returns the cached path with a warning. This keeps workers running across transient network failures and on permanently air-gapped sites. If the cache is empty or corrupt,
RuntimeErroris raised.
Cache integrity: SHA-256 checksums are verified on every ensure_model call when a checksum is recorded — on cold start, after every download, during disk reconciliation, and on every warm-cache hit. There is no shortcut. For multi-gigabyte checkpoints this is the dominant cost of the call, but ensure_model only runs when a worker file changes (rare) or the worker container restarts; in exchange we get unconditional bit-rot detection. A checksum mismatch on a downloaded artifact triggers a re-download attempt; a download whose checksum does not match the catalog is rejected and the partial file removed.
Pre-staging weights for air-gapped deployments
On a site without internet access, an operator can place weights directly into the cache:
mkdir -p ~/.cyberwave/models/yolov8n
cp /usb-stick/yolov8n.pt ~/.cyberwave/models/yolov8n/
Edge Core picks up the file on the next ensure_model("yolov8n") call, computes a SHA-256, and writes a sidecar metadata.json so subsequent runs are deterministic. The runtime field is inferred from the file extension (.pt → ultralytics, .onnx → onnxruntime, .engine/.trt → tensorrt, .tflite → tflite, .pth → torch, .xml → opencv); provide a hand-written metadata.json to override.
Updating in place. Operators can drop a new build into the same directory and Edge Core will detect the change on the next call:
cp /usb-stick/yolov8n-v2.pt ~/.cyberwave/models/yolov8n/yolov8n.pt
The mismatch between the on-disk SHA-256 and the manifest checksum triggers a re-stamp (not a re-download), provided the sidecar still records downloaded_from: prestaged. This keeps offline edges functional across model upgrades. Files that were previously downloaded by Edge Core keep the corruption-detection semantics — bit-rot still triggers a re-download attempt rather than being silently accepted.
Pre-staged files are never auto-overwritten by catalog updates. Once a file lives under cache_dir/{model_id}/ with a downloaded_from: prestaged sidecar, Edge Core treats it as the source of truth and skips the catalog probe entirely. To force a re-download from the Cyberwave catalog, evict the model:
rm -rf ~/.cyberwave/models/yolov8n
Provide a hand-written metadata.json (with filename, checksum_sha256, and runtime) when there are multiple weight files in the directory or when corruption detection should compare against a known-good hash.
Self-healing the model cache
A failed download leaves an empty cache_dir/{model_id}/ directory behind — the mkdir on the download path runs before the network fetch, and any error in the downloader (HTTP timeout, missing upstream weight, runtime self-download crash) skips the cleanup. Without intervention, the SDK in the worker container would later route that directory into torch.load and crash with IsADirectoryError on every restart.
Edge Core handles this in three layers:
- Per-download cleanup (conservative). If
_download_modelor_download_runtime_managedraises, the staging directory is removed before the exception propagates, provided it contains only an.dl_*.partpartial-download file, ametadata.jsonsidecar, or nothing at all. Operator-staged content blocks the prune. - Startup sweep (conservative).
ModelManager()walkscache_dir/once at construction time and prunes any per-model subdirectory that is not tracked in the manifest and contains only orphan cruft. This catches hosts that were already wedged before this code shipped. Operator-staged content blocks the prune. evict_model('foo')(decisive). When there is no manifest entry butcache_dir/foo/exists on disk, the directory is removed unconditionally andTrueis returned. This is the recovery hook for a remote eviction API to call without shell access to the host; an explicitevictis treated as a stronger operator-intent signal than the conservative auto-sweeps.
For the two conservative layers, a directory containing an unexpected file (an operator's README.md, a hand-staged weight file with an unusual name, a sub-directory) is preserved untouched — "human always wins". Use evict_model explicitly when you mean to clear the slot regardless of contents.
ML runtime extras
The default pip install cyberwave-edge-core keeps the runtime surface lean — no Torch, no Ultralytics, no ONNX. That's enough for workflows that resolve weights through a backend-hosted URL or pre-staged files. Catalog entries that rely on the runtime-managed download fallback (step 5.3 above) need the matching runtime installed on the edge host:
# Pull in Ultralytics so YOLO/YOLOE .pt entries can self-download
pip install 'cyberwave-edge-core[ml]'
# Other available extras (mirror the cyberwave SDK extras 1:1):
pip install 'cyberwave-edge-core[ml-onnx]' # onnxruntime
pip install 'cyberwave-edge-core[ml-tflite]' # tflite-runtime
pip install 'cyberwave-edge-core[ml-wakeword]' # openwakeword + onnxruntime
pip install 'cyberwave-edge-core[ml-stt]' # pywhispercpp (Whisper.cpp GGML)
pip install 'cyberwave-edge-core[ml-stt-faster]' # faster-whisper (CTranslate2)
pip install 'cyberwave-edge-core[ml-audio]' # silero-vad + torchaudio (Audio Assistant)
pip install 'cyberwave-edge-core[fuzzy-match]' # rapidfuzz (Fuzzy Matcher)
pip install 'cyberwave-edge-core[microphone]' # host mic capture (Audio Track trigger)
pip install 'cyberwave-edge-core[ml-all-vision]' # YOLO / ONNX / TFLite only
pip install 'cyberwave-edge-core[ml-all-audio]' # all speech extras (avoid on Pi — pick ml-stt / ml-stt-faster per model)
pip install 'cyberwave-edge-core[ml-all]' # CI/dev fat bundle only
When a required runtime is missing, the resolver raises a RuntimeError that spells out the three workarounds: drop the weight file into ~/.cyberwave/models/{model_id}/, upload to /api/v1/mlmodels/{uuid}/weights, or set metadata.download_url on the catalog entry.
On a frozen PyInstaller binary (cyberwave edge install --channel dev|staging), sys.executable is the edge-core CLI wrapper rather than a Python interpreter, so the runtime-managed downloaders cannot spawn the subprocess they need. In that case ensure_model raises _RuntimeManagedDeferred, which ensure_models treats as non-fatal: the model is omitted from last_ensure_failures and the worker container — which bundles the runtime — resolves the weight on its first inference call. This prevents a misleading model_download_failure alert from firing while the workflow runs fine (CYB-2103).
Worker container
Edge Core manages one ML worker container per edge device (container name: cyberwave-worker-{env_uuid[:8]}). The worker container runs Python worker scripts from the local workers directory and has access to cached model weights.
Worker directory layout
Place worker scripts in {config_dir}/workers/ (default: ~/.cyberwave/workers/):
~/.cyberwave/
├── workers/
│ ├── detect_people.py # Custom worker
│ └── cyberwave.yml # Optional: list model requirements
└── models/ # Auto-managed model cache
├── manifest.json
└── yolov8n/
└── yolov8n.pt
cyberwave.yml
Optionally declare model requirements so Edge Core can pre-download them before starting the worker container:
models:
- yolov8n
- background-subtraction
Edge Core also auto-detects models by scanning cw.models.load("...") calls in worker Python files.
Worker container environment variables
| Variable | Value |
|---|---|
CYBERWAVE_API_KEY |
Injected from credentials |
CYBERWAVE_ENVIRONMENT_UUID |
Active environment UUID |
CYBERWAVE_TWIN_UUIDS |
Comma-separated twin UUIDs in environment |
CYBERWAVE_DATA_BACKEND |
zenoh |
ZENOH_CONNECT |
Set when a Zenoh router is configured |
ZENOH_SHARED_MEMORY |
false by default (opt-in; requires --ipc=host) |
File watching and hot-reload
Edge Core monitors {config_dir}/workers/ every reconcile cycle (~15 seconds). When .py files are added, removed, or modified, Edge Core automatically:
- Re-scans model requirements.
- Pre-downloads any missing models.
- Restarts the worker container with the updated files.
A minimum cool-down of 10 seconds between successive automatic restarts prevents rapid churn when files are written incrementally (e.g. by rsync or scp).
Workflow-driven worker lifecycle
The worker container is brought up and torn down based on whether any active workflows are currently synced for the connected twins:
- Startup: after pulling worker files from the backend (step 8), Edge Core inspects
{config_dir}/workers/. If at least onewf_*.pyfile is present, the worker container is started; otherwise it is left down — and thecyberwaveos/edge-ml-workerimage is not pulled. - Periodic reconcile: every ~5 minutes (configurable via
CYBERWAVE_WORKER_SYNC_INTERVAL_LOOPS), Edge Core resyncs worker files from the backend. If a workflow was activated mid-run and new files appeared, the worker container is started. If every workflow was deactivated and the directory is now empty, the container is stopped. Both calls are idempotent. - Immediate reconcile on activate: when a
run_on_edgeworkflow is activated (UI, CLI, or API), the backend publishes async_workflowscommand oncyberwave/twin/{twin_uuid}/commandfor each twin the workflow references. Edge Core runsreconcile_worker_syncright away so the newwf_*.pylands within seconds instead of up to one periodic interval. When the worker container is already running, Edge Core also bounces it viaWorkerManager.restartso the new modules are actually imported —start()is a no-op for a running container, so without this step activation would only take effect after the next file-watch tick (or a manualcyberwave worker restart). The bounce is skipped when the container itself started after the newest change in{config_dir}/workers/— it already imported those files on the way up, so restarting would only discard a boot in progress and delay the workflow going live. Anything that cannot be determined (a faileddocker inspect, an unreadable workers directory) falls back to bouncing. Failures fall back to the periodic reconcile; the MQTT nudge is best-effort, not a correctness guarantee. - Immediate reconcile on remove: when a
run_on_edgeworkflow is deactivated, soft-deleted, or has itsrun_on_edgeflag flipped off, the backend publishes a surgicalremove_workflow_workercommand on the same twin command topic, naming the specific worker filename(s) to unlink. Edge Core unlinks the matchingwf_*.pyfrom{config_dir}/workers/and immediately reconciles the worker container lifecycle, so deactivation propagates within seconds rather than waiting up to one periodic interval (~5 minutes) plus the bulk-sync two-strikes cleanup cycle. For partial deactivations (sibling workflows still active), the same in-place restart fires so the removed worker is actually unloaded. The same nudge also covers workflows soft-deleted as part of a workspace or environment teardown. The periodic reconcile remains the correctness backstop if the MQTT publish is dropped. - MQTT command dedupe: both
sync_workflowsandremove_workflow_workercarry arequest_id; Edge Core keeps a bounded FIFO of the last 1024 IDs it has handled per process and drops repeats. QoS retries and broker re-publishes therefore don't spawn duplicate sync threads or repeat unlink operations. - Sync errors: if a sync cycle reports any errors, the lifecycle reconcile is skipped to avoid churning a healthy worker on transient API failures, and stale-file cleanup is skipped too. A twin whose fetch failed reports no workflows, which is indistinguishable from a genuine deactivation, so deleting on that picture would let an outage, an expired token or a DNS blip spanning two sync periods unlink every
wf_*.pyon the device. Deletion needs a complete picture; the next successful sync re-evaluates state.
How this is tested
Reconcile branching, watcher debouncing, and MQTT command handling are unit-tested with mocked Docker in tests/test_worker_lifecycle_reconcile.py, tests/test_worker_watcher.py, tests/test_worker_health.py, and tests/test_startup_remove_workflow_worker.py. ResourceLimits argv assembly is covered in tests/test_worker_health.py (TestResourceLimits).
Tier 1 — scripted lifecycle integration (tests/integration/test_worker_lifecycle_docker.py, marker: docker): 14 deterministic phases exercise the real Docker daemon — cold start, add/remove worker files, restart-in-place, surgical remove, edge-core process restart, cooldown enforcement, image pull policy, request_id dedupe for both remove_workflow_worker and sync_workflows, health-monitor record_stop wiring, spontaneous-exit detection, hot-reload through WorkerWatcher.force_restart, and a full activate→deactivate→reactivate operator cycle. The suite auto-skips when Docker is unavailable and uses a session-built cyberwave-worker-stub image so it never pulls the real cyberwaveos/edge-ml-worker.
Tier 2 — Hypothesis chaos (tests/integration/test_worker_lifecycle_chaos.py, marker: docker): a RuleBasedStateMachine drives the same surface (WorkerManager, reconcile_worker_lifecycle, _run_remove_workflow_worker, WorkerWatcher) with randomized operation interleavings against the real Docker daemon, asserting eight cross-cutting invariants after every step (state consistency, no spurious docker rm, container-name stability, request_id dedupe, immutable-tag pull-at-most-once, record_stop count matches running→exited transitions, cooldown respected, no orphan containers). PR CI runs with max_examples=20, stateful_step_count=30, derandomize=True (deterministic, ≤ 3 min); staging also runs a lighter chaos slice (max_examples=5, stateful_step_count=15). The nightly workflow (.github/workflows/edge-core-nightly-chaos.yml) mirrors the PR test_edge_core job (Tier 1 Docker integration, Tier 3 + Tier 3-ML shell E2E against a live backend) and adds Tier 2 chaos with max_examples=200, stateful_step_count=100 and a random seed; it files a GitHub issue on failure without blocking PRs. Monotonic timing for cooldown and health windows goes through cyberwave_edge_core._clock.now_monotonic() so tests can advance time without sleeping.
Tier 3 — Full E2E against a live backend (cyberwave-edge-nodes/test_worker_lifecycle_e2e.sh): closes the gaps left by Tier 1/2 by exercising the complete workflow activation pipeline: creates real backend objects (project → environment → twin → workflow with manual-trigger + send-MQTT nodes) via the HTTP API, activates the workflow, and asserts that the sync_workflows MQTT message is delivered to a running edge-core, that edge-core fetches worker_source from the /edge-sync endpoint, writes the resulting wf_*.py file, and starts the worker container. Deactivation is then exercised and the container stop is asserted. Runs after test_edge_core_and_cli.sh in the same CI job so the backend and broker are already healthy. The test is self-contained: it builds a minimal cyberwave-worker-stub:it (Alpine sleep infinity) and a cyberwave-edge-test image, and tears both down on exit.
Tier 3-ML — Model download and inference E2E (cyberwave-edge-nodes/test_ml_inference_e2e.sh): extends the Tier 3 test with two focused phases. Phase A creates a camera_frame → call_model(yolov8n.onnx) workflow, activates it, and asserts that edge-core pre-downloads yolov8n.onnx (~6-12 MB, YOLOv8 Nano ONNX from the Cyberwave CDN) via ModelManager.ensure_models() before starting the worker container. Phase B starts cyberwaveos/edge-ml-worker:local and runs onnxruntime inference on a synthetic 640×640 image, asserting clean exit and a valid 3-D output tensor. Requires cyberwaveos/edge-ml-worker:local to be pre-built (see the script header for the docker buildx build command); the CI step skips gracefully when the image is not available.
Worker image refresh policy
WorkerManager._ensure_image_pulled decides whether to issue docker pull before each worker (re)start:
Tag basename (with optional -gpu/-cpu/-arch suffix) |
Mutability | Pull behaviour |
|---|---|---|
latest, dev, staging, nightly, edge, main, master |
Mutable | Pull every time, even when the image is already present locally. If the registry is unreachable but a local copy exists, fall back to the local copy and warn. |
Anything else (v1.2.3, dated build IDs, @sha256:…, local) |
Immutable | Skip the pull when the image is already present locally; only pull when missing. |
This avoids the previous failure mode where a stale cyberwaveos/edge-ml-worker:dev-gpu image stayed cached after a developer pushed a new build — operators no longer need to remember to docker rmi before restarting the worker. Immutable tags keep the original fast-path so versioned production deployments are not slowed down by an extra round-trip to the registry on every restart.
local counts as immutable because nothing publishes it: a :local image is whatever the last local docker build produced, so a pull can only fail — a wasted round-trip, or a connect timeout on an offline machine. A :local tag with no local copy is still pulled, like any other missing image.
Driver images follow a narrower rule. They are pulled on every start unless the tag is local, local-* or *-local and the image is already on the host (see Local driver builds are never pulled). The reason is the same: a local build must survive a restart.
Pinning a custom worker image (CYBERWAVE_WORKER_IMAGE)
resolve_worker_image() consults CYBERWAVE_WORKER_IMAGE before falling back to the CYBERWAVE_ENVIRONMENT-derived tag. This is the worker-side counterpart to the driver_overrides field in credentials.json for camera/asset drivers, and is the recommended way to run a locally-built or hot-patched worker image without re-pushing to the registry.
Two ways to set it (both honoured by get_runtime_env_var):
-
Operator config (preferred for dev hosts, no
sudo) — add to theenvsblock of~/.cyberwave/credentials.json:{ "envs": { "CYBERWAVE_ENVIRONMENT": "dev", "CYBERWAVE_WORKER_IMAGE": "cyberwaveos/edge-ml-worker:local" } }
-
Systemd dropin (preferred for managed deployments) —
sudo systemctl edit cyberwave-edge-core:[Service] Environment=CYBERWAVE_WORKER_IMAGE=cyberwaveos/edge-ml-worker:local
Either way, restart edge-core (sudo systemctl restart cyberwave-edge-core) to reload the resolver.
Use the :local tag (no -gpu/-cpu/-jetson/-hailo suffix) — same convention as the camera-driver :local tag. _run_container auto-appends the accelerator suffix (-jetson on Jetson hosts, -gpu on discrete-NVIDIA hosts, -hailo on Pi 5 + AI HAT+) for cyberwaveos/edge-ml-worker:* overrides, so the operator only commits and pins one tag.
Typical hot-fix loop (mirroring the camera-driver :local tag pattern):
# 1. Hot-patch the running container (e.g. swap an SDK runtime file).
WORKER=cyberwave-worker-<fingerprint>
docker cp /path/to/patched/file.py "$WORKER:/usr/local/lib/python3.12/dist-packages/cyberwave/.../file.py"
docker exec -u root "$WORKER" rm /usr/local/.../file.cpython-312-x86_64-linux-gnu.so
# 2. Snapshot the patched container as a local-only tag (and an alias on the
# accelerator-suffixed tag matching the host, so the same image is used
# regardless of which path `_run_container` resolves to on this host).
docker commit "$WORKER" cyberwaveos/edge-ml-worker:local
docker tag cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu
# On Jetson hosts, alias -jetson instead of -gpu:
# docker tag cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-jetson
# 3. Tell edge-core to use it (see the two options above), then restart it.
# Pull will fail (registry has no :local) and `_ensure_image_pulled`
# falls back to the locally-present image.
sudo systemctl restart cyberwave-edge-core
The patched image survives every docker rm/docker run cycle from edge-core's reconcile loop. To revert, remove the env var (or systemctl revert cyberwave-edge-core if you used the dropin) and docker rmi cyberwaveos/edge-ml-worker:local cyberwaveos/edge-ml-worker:local-gpu (or …local-jetson on Jetson hosts).
When overriding to a registry outside cyberwaveos/edge-ml-worker:* (e.g. an internal mirror), include the accelerator suffix (-gpu / -jetson / -hailo) yourself if you need accelerator access — the auto-suffix path in _run_container only triggers for the canonical cyberwaveos/edge-ml-worker: prefix.
Worker startup probe
After docker run --detach, Edge Core polls the container status for up to 30 seconds (configurable via CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS) waiting for it to reach running state. The probe uses docker inspect to check the full container state, including restart count:
running: success — container is up.exited/deadwith no prior restarts: definitive failure — an alert is sent to all linked twins.exitedwithRestartCount > 0: Docker's restart policy (--restart=unless-stopped) is active — the probe keeps waiting.created/restarting/other transient states: Docker is still setting up the container — the probe keeps waiting.- Probe window elapsed: the container may still come up (Docker keeps trying via its restart policy), so Edge Core returns success and sends a
worker_start_failurealert for visibility. The health monitor and periodic reconcile loop track the eventual outcome.
This resilient approach ensures that slow container startups on resource-constrained devices (e.g. Raspberry Pi) do not block the rest of Edge Core's boot sequence.
Worker readiness
A running container is not yet a usable worker. The process inside still has to import your worker modules (downloading model weights the first time), connect to MQTT, subscribe its hooks and run warm-up inference — minutes on a cold device.
The worker reports each of those milestones to /run/cyberwave/worker-status.json, which Edge Core mounts into the container and reads once per reconcile cycle (every 15 s). The worker_starting alert stays open across the whole sequence, so the dashboard shows Loading models or Warming up models — 2 of 5 models instead of going quiet the moment the container starts, and a workflow run dispatched to the device explains what it is waiting for rather than appearing stalled.
ready: the alert resolves.- Container exits during startup: the alert reports the failure and the phase it reached.
CYBERWAVE_WORKER_READY_TIMEOUT_SECONDSelapsed (default 15 minutes): the alert reports that the worker took too long to load its models.- No status file within 60 seconds: the worker image predates status reporting, so Edge Core falls back to treating a running container as started.
The readiness timeout is measured from the container's own start timestamp rather than from when Edge Core began watching, so a long image pull does not eat into the worker's own startup budget and a container restart resets the clock by itself.
The progress card does not survive an Edge Core restart, though. A graceful restart stops the worker container first, so the next start opens a fresh card. But if Edge Core is killed while the worker container keeps running — an out-of-memory kill, a watchdog reset, a crash loop — the card is resolved on the next boot and is not re-opened for that container, and the worker finishes loading with nothing on screen.
cat /run/cyberwave/worker-status.json on the device shows the current phase directly. The file lives on tmpfs, so it costs no flash writes and does not survive a reboot.
Worker health monitoring
Edge Core continuously monitors the worker container for spontaneous exits and crash loops:
-
Restart accounting: every restart is recorded with a timestamp and reason.
-
Sliding-window rate limiting: if more than 5 restarts occur within 5 minutes, the circuit-breaker trips and automatic restarts are suppressed. The breaker resets automatically once the window clears.
-
Spontaneous exit detection: if the container exits without a deliberate restart, a warning is logged so operators can investigate. The detector recognises deliberate stops via two channels:
- Callers that hold the monitored
WorkerManager(e.g.WorkerManager.stop) notify it viarecord_stop(). The steady-state workflow deactivation path (reconcile_worker_lifecycle) reuses the watcher's monitored manager so this channel covers it directly. - For cross-instance stops driven by managers without the shared monitor (the "Restart edge core" flow's
_stop_worker_container_for_restart, and cold-boot MQTT command handlers that fire before the watcher's first tick), the monitor consults an "expected to be running" probe that returns False when either the workers directory is empty or an edge-core restart is in flight. Both cases downgrade the log to INFO.
The "may indicate a crash loop" wording is reserved for exits that look unexpected — real crashes, OOM kills, or manual
docker stopwhile workflows are active. - Callers that hold the monitored
Use cyberwave-edge-core worker health to inspect the full restart history and circuit-breaker state.
Resource limits
You can constrain the worker container's CPU and memory usage by setting CYBERWAVE_WORKER_CPU_QUOTA_PERCENT and CYBERWAVE_WORKER_MEMORY_MB environment variables on the edge host (both optional). When set, Edge Core passes the corresponding --cpu-quota, --cpu-period, and --memory flags to docker run.
GPU memory fraction can be limited via CYBERWAVE_GPU_MEM_FRACTION (a float between 0 and 1); this is passed as an env var into the worker container.
High availability and watchdog
Edge Core includes several layers of resilience for edge devices running ML inference workloads (e.g. Raspberry Pi 4 with YOLO). These features prevent the orchestrator from dying under heavy CPU or memory load and ensure automatic recovery when failures occur.
There are four independent watchdog/supervision layers, each covering a different failure mode:
| Layer | What it watches | Mechanism | Recovery action |
|---|---|---|---|
| Hardware watchdog | Kernel hang, total board lockup | /dev/watchdog ping |
SoC resets the board |
| Systemd watchdog | Orchestrator process hang/exit | Type=notify + WatchdogSec |
systemd restarts the service |
| Worker container | ML worker container health | WorkerHealthMonitor + circuit breaker |
Docker restart / circuit-break |
| Driver containers | Per-driver containers that flap, exit or disappear | reconcile_driver_* in startup.py |
restart suppression on flap; revival of clean exits, and of removed containers after a grace period |
Note: these are infrastructure watchdogs. They are distinct from the in-process command/teleop watchdogs in cyberwave-edge-nodes and cyberwave-edge-runtime, which stop driver motion if no commands arrive within a window — the term "watchdog" is overloaded.
Systemd watchdog
The systemd unit generated by cyberwave edge install configures WatchdogSec=60 and Type=notify. Edge Core pings the systemd watchdog every reconcile cycle (~15 s). If the process hangs or dies, systemd restarts it automatically. The unit uses Restart=always (which is the systemd equivalent of Docker's unless-stopped — systemd skips the restart when the operator runs systemctl stop). always is preferred over on-failure for robustness: both restart on SIGKILL/OOM, but only always also restarts on an accidental exit(0) from a buggy code path.
Startup readiness uses a two-phase handoff (ProcessWatchdog.mark_ready() → ProcessWatchdog.start_pinging()): READY=1 is sent unconditionally as soon as the blocking boot work (driver images pulled, MQTT connected, twin sync reconciled) is done, while the periodic WATCHDOG=1 pings — which sd_notify(3) restricts to the $WATCHDOG_PID process — are driven from the runtime loop in the same process. The split is what makes the standalone PyInstaller binary work under Type=notify: the --onefile bootloader holds MainPID (and therefore $WATCHDOG_PID) while the actual Python app runs in a forked child whose PID differs.
During long-running startup operations (driver image pulls and ML worker preparation, including model downloads and the worker image pull), Edge Core sends EXTEND_TIMEOUT_USEC notifications via sd_notify every 5 seconds. This extends the systemd start timeout while preparation is active, including periods without pull progress events, so TimeoutStartSec does not kill the service during a slow download, but not without limit: extensions stop when the startup budget (30 minutes by default) has passed since the watchdog was created, and they stop at once on SIGTERM (see Bounded startup and shutdown below). A STATUS= notification is also sent so that systemctl status shows which images are being pulled. On SIGTERM, startup checks stop advancing to new drivers or workers. An in-flight Docker operation may still finish or reach systemd’s normal stop timeout; it cannot prolong shutdown with startup heartbeats.
Because the watchdog pings come from the runtime loop, anything that blocks the loop for a minute gets the service killed, and an SDK REST call made without a timeout could do exactly that when the backend never answered. Before the runtime loop starts, main.py therefore calls bound_sdk_request_timeouts(), which patches the generated SDK client so that every request without its own timeout gets one of 10 s (SDK_REQUEST_TIMEOUT_SECONDS). Requests that already pass a timeout keep it, the patch is applied once however often it is called, and boot-time calls made before it are not affected. The bound applies to each attempt, not to the call: urllib3's default policy retries a timed-out GET three times, so one GET can still take about 40 s. Nor does it bound a sequence of calls: a driver revival pass makes one GET for the twin list and up to two per linked twin, one after another. The trade-off is deliberate: a legitimately slow REST call from the runtime loop now fails and is retried on a later cycle, instead of taking the whole service down with it.
Bounded startup and shutdown
Startup keeps Type=notify readiness and a base TimeoutStartSec=900. The pull heartbeat above can extend it, but only up to the startup budget, counted from the watchdog's creation, so a startup that never succeeds cannot stay "activating" forever. The budget is 30 minutes; an edge whose first start pulls several gigabytes over a slow link can raise it with CYBERWAVE_STARTUP_TIMEOUT_MINUTES, a positive whole number of minutes (any other value is ignored with a warning). Set it in the envs of credentials.json: the service reads it from there, and so does the unit generator, which derives JobTimeoutSec from it and deliberately ignores the shell it runs in, since the service never sees that shell. After changing it, run sudo cyberwave edge install again, or let the next package upgrade regenerate the unit, before restarting the service. SIGTERM calls ProcessWatchdog.begin_shutdown(): it sends STOPPING=1 and refuses every later extension, READY=1 and watchdog ping, even when the signal arrives in the middle of the startup checks, which then exit through the normal shutdown path without announcing readiness. A consequence worth knowing: /dev/watchdog is never opened after SIGTERM, so a stop can never arm the hardware reset.
The generated unit also bounds the other end. TimeoutStopSec=90 gives the orchestrator 90 seconds to stop before systemd kills what is left in the service's control group (KillMode=control-group, SendSIGKILL=yes), and JobTimeoutSec in the [Unit] section bounds the start job as a whole, including time spent waiting behind its dependencies, five minutes beyond the startup budget (35min by default). Killing the orchestrator does not stop Docker containers by itself, so CLI uninstall checks separately that the service stopped and removes the containers before deleting configuration.
The CLI side matches: on Linux, cyberwave edge start, restart and install submit the systemd job without waiting for readiness and print the commands to follow it, while stop and uninstall wait up to 120 seconds for the service to stop and, if they cannot confirm it, report the failure and keep the configuration. Deploy both packages to get both halves. The media and lifecycle investigation has recovery steps, limits and verification commands.
Hardware watchdog (Raspberry Pi)
On Linux devices with /dev/watchdog (all Raspberry Pi models), Edge Core optionally drives the hardware watchdog timer. If the edge-core process stops pinging for longer than the hardware timeout, the SoC resets the board. This is a last-resort safety net for kernel-level hangs or total process death.
Edge Core asks for twice the unit's WatchdogSec, and at least 120 seconds, so a hung process is first restarted by systemd and only a hang that outlives that reboots the board. Override it with CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS. A driver may round or clamp the request, and some refuse it outright and keep their own timeout: a Raspberry Pi's bcm2835_wdt tops out at 15 seconds. The log line Hardware watchdog enabled (requested timeout=…, effective=…) shows the timeout actually running, and a warning follows when it is not above WatchdogSec.
Disable with CYBERWAVE_HARDWARE_WATCHDOG=false.
OOM protection
Edge Core lowers its OOM score (oom_score_adj=-800) at startup so the Linux OOM killer preferentially terminates the heavier ML worker container (which Docker restarts automatically via --restart unless-stopped) instead of the orchestrator. The systemd unit reinforces this with OOMScoreAdjust=-800.
Auto-detected memory limits
On devices with ≤4 GB total RAM, Edge Core automatically applies a memory limit to the worker container, reserving ~25% of total memory for the OS and edge-core. This prevents the worker from consuming all available memory and triggering the OOM killer. Explicit CYBERWAVE_WORKER_MEMORY_MB values override the auto-detected limit. Disable auto-detection with CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT=false.
Resource monitoring
SystemResourceMonitor (resource_monitor.py) watches four host gauges: memory in use, from /proc/meminfo; CPU load, the busy share of all cores between two readings of /proc/stat; disk usage of the filesystem that holds Docker's data root; and the temperature of the hottest CPU thermal zone. The runtime loop calls check() on every tick, but the monitor reads the host at most every 30 s (MONITOR_CHECK_INTERVAL_SECONDS) and otherwise hands back its last snapshot, so the CPU figure is a 30 s average and is absent from the first snapshot. Memory at 85 % or a temperature of 75 °C logs Resource pressure elevated at INFO. Memory at 92 % or 82 °C logs CRITICAL resource pressure as a warning and increments consecutive_critical, which drops back to zero on the first reading below both critical thresholds. Each message is rate-limited to one a minute.
CPU, memory and disk also raise an alert when they saturate. reconcile_resource_saturation runs after check() and looks at each new snapshot once: two snapshots in a row at or above CYBERWAVE_RESOURCE_SATURATION_PERCENT (95 % by default; an invalid value logs a warning and falls back to 95) mark the resource saturated, and two in a row below a level five points lower (90 % by default) clear it, so a single spike or dip changes nothing. On the way up Edge Core runs docker stats --no-stream once to name the busiest containers, logs Edge CPU saturated: 97% (threshold 95%) for 30 s; top: cyberwave-driver-658d581b 61%, …, repeats that line every minute while the resource stays saturated, and sends an edge_cpu_saturation, edge_memory_saturation or edge_disk_saturation warning to every twin whose driver it manages. The alert is raised once per crossing, so the percentage in its description does not defeat the backend's dedupe; its metadata carries resource, percent and threshold. A twin the alert could not reach is retried on each later snapshot with the same content until it is delivered or the saturation clears. On the way down the alerts are resolved, and a resolve that fails is retried on the next snapshot. While the backend is unreachable, no alert is raised or resolved: the first call that fails pauses every cloud call of Edge Core, and what is left waits for a snapshot after the pause. The saturation state lives in memory, so a new process cannot know what the last one raised: when the runtime loop starts it treats every resource as possibly alerted, resolves those alerts on every managed twin at the resource's first snapshot below the clear level, and lets a resource that saturates first raise again, which the backend folds into the alert still active. An edge that manages no driver only logs. Temperature keeps its own thresholds and raises nothing.
Board power draw is sampled separately, from sysfs (Jetson INA3221 in_power0_input, generic hwmon/power1_input, or power_supply/BAT*/power_now, in that order), and fresh on every reading rather than from the 30 s snapshot, because draw can change several-fold between idle and motion in under a second. It is published as power_mw by the same two transports as the other gauges and shows up as the pwr X.XW pill in the Edge details panel of the dashboard.
Set CYBERWAVE_BATTERY_WH to your battery pack's watt-hour capacity to enable a rough runtime estimate on the dashboard (e.g. 36.0 for a 36 Wh pack). The estimate is a simple linear model (battery_wh / current_watts hours); it ignores discharge curves and temperature effects and is only meant to answer "will this run another 30 minutes?".
| Variable | Default | Description |
|---|---|---|
CYBERWAVE_HARDWARE_WATCHDOG |
true |
Enable/disable the hardware watchdog (/dev/watchdog) |
CYBERWAVE_HARDWARE_WATCHDOG_TIMEOUT_SECONDS |
twice WatchdogSec, at least 120 |
Hardware watchdog timeout requested from the driver. Keep it above WatchdogSec. See Hardware watchdog. |
CYBERWAVE_WORKER_AUTO_MEMORY_LIMIT |
true |
Enable/disable auto-detected memory limits on ≤4 GB hosts |
CYBERWAVE_WORKER_STARTUP_PROBE_SECONDS |
30 |
Seconds to wait for the worker container to reach running state after docker run |
CYBERWAVE_WORKER_READY_TIMEOUT_SECONDS |
900 |
Seconds to wait for the worker process to finish starting (load and warm up its models) once its container is running |
CYBERWAVE_WORKER_STATUS_PATH |
/run/cyberwave/worker-status.json |
Where the worker writes its startup milestones. Set by Edge Core on the worker container to match the directory it mounts; override only if /run is unavailable |
CYBERWAVE_BATTERY_WH |
(unset) | Battery pack capacity in watt-hours. Enables the battery uptime estimate in the Edge details panel of the dashboard. |
CYBERWAVE_RESOURCE_SATURATION_PERCENT |
95 |
CPU, memory and disk usage at which Edge Core raises the edge_*_saturation alerts; they clear five points lower. See Resource monitoring. |
CYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T |
(unset) | Set to 1 to use the -jetson worker image and NVIDIA runtime on a Jetson below L4T r38 (JetPack 7). See the accelerator precedence list below. |
GPU support
When an NVIDIA container runtime is detected (docker info reports nvidia runtime), Edge Core adds --gpus all to the worker container's docker run command.
For the worker image, Edge Core also rewrites the tag to point at the accelerator-specific sibling. The precedence is:
-
Jetson host on JetPack 7 (
/etc/nv_tegra_releasereports L4T r38 or newer, orCYBERWAVE_PLATFORM_VARIANT=jetson) →cyberwaveos/edge-ml-worker:<tag>-jetson+--runtime nvidia. Wins over the plain-gpubranch: on Jetson the nvidia-container-runtime is present too, but the amd64-gpublob has the wronglibcuda.soABI (desktop/server CUDA vs Tegra CUDA). -
Jetson host on JetPack 6 or older (L4T r36 and below) → plain
cyberwaveos/edge-ml-worker:<tag>, no GPU passthrough, with a warning. The-jetsonimage's CUDA 12.9 wheels cannot load against a JetPack 6 driver — cuBLAS fails on the first matmul while convolutions still succeed, so the model loads and then detects nothing. The plain tag is multi-arch, so arm64 CPU inference works on the same host. A-jetsontag pinned viaCYBERWAVE_WORKER_IMAGEis demoted the same way; a pin to an image outsidecyberwaveos/edge-ml-workerkeeps the NVIDIA runtime, since the floor only speaks for our own build. SetCYBERWAVE_JETSON_ALLOW_UNSUPPORTED_L4T=1to force the GPU path anyway (e.g. for a hand-built JetPack 6 image).The choice is made when the worker container is created. A board that already has a
-jetsonworker running keeps it across an Edge Core upgrade — startup skips image selection entirely when the container is already up, and its--restart unless-stoppedpolicy survives reboots. Rundocker rm -f cyberwave-workeronce to pick up the new selection. -
NVIDIA container runtime detected (not Jetson) →
cyberwaveos/edge-ml-worker:<tag>-gpu+--gpus all. -
/dev/hailo0present and no GPU runtime →cyberwaveos/edge-ml-worker:<tag>-hailo+ device passthrough. -
None → CPU base image.
If the accelerator-specific tag cannot be pulled (e.g. the sibling isn't published for your channel yet), Edge Core falls back to the base tag and logs the demotion — the worker still comes up, just on CPU.
Driver GPU passthrough is opt-in via asset metadata. When a driver's config includes "prefer_gpu": true and the host has:
- NVIDIA container runtime available (
docker inforeportsnvidia), and nvidiaset as the default runtime in/etc/docker/daemon.json
…Edge Core passes --gpus to the driver container. The optional "gpu" field controls which GPUs are exposed:
gpu value |
Docker flag | Use case |
|---|---|---|
| (not set) | --gpus all |
All available GPUs (default) |
1 |
--gpus 1 |
Limit to 1 GPU |
"device=0,2" |
--gpus "device=0,2" |
Specific GPU devices |
Example driver metadata:
{
"drivers": {
"default": {
"docker_image": "cyberwaveos/go2-ros2-driver:humble",
"prefer_gpu": true,
"gpu": "all"
}
}
}
If the NVIDIA runtime is available but not configured as the default in daemon.json, Edge Core logs an informational message with setup instructions instead of silently skipping GPU passthrough.
Jetson detection
Edge Core auto-detects NVIDIA Jetson hardware via /etc/nv_tegra_release, and reads the L4T major release (R36 = JetPack 6, R38 and up = JetPack 7) from its first line. The release gates the ML worker image only — the jetson- driver images below build from the same ROS base as their plain counterparts and differ only in a resource profile (smaller pointcloud budget, software H.264, capped publish rates), so they link no CUDA and carry no L4T floor. When running on a Jetson:
- The platform key
linux-aarch64-jetsonis added to the driver resolution order, allowing asset metadata to specify a Jetson-optimised image. - If no
linux-aarch64-jetsondriver key exists in metadata, Edge Core rewrites the image tag by prependingjetson-(e.g.cyberwaveos/go2-ros2-driver:humble→cyberwaveos/go2-ros2-driver:jetson-humble). If the Jetson-prefixed image is not available, it falls back to the original tag automatically.
Override detection with CYBERWAVE_PLATFORM_VARIANT=jetson for testing.
Multi-camera orchestration
When multiple cameras are connected to the same edge device (each represented as a separate digital twin), Edge Core orchestrates them as follows:
-
One driver per camera: Each camera twin gets its own
cyberwave-driver-{uuid[:8]}container. Child camera twins that are attached to a parent twin share the parent's driver instead. -
One shared worker: A single
cyberwave-worker-{env[:8]}container receives frames from all cameras. The worker container receivesCYBERWAVE_TWIN_UUIDSas a comma-separated list of all linked twins. -
Readiness probes: Edge Core waits for all driver containers to reach a
runningstate before starting the worker. If some drivers fail, the worker starts anyway so healthy cameras can be utilized. -
Model pre-download: Before the worker starts, Edge Core scans worker scripts and pre-downloads all referenced ML models.
-
Driver health monitoring: when a driver container that was running is found stopped or removed,
reconcile_driver_health_for_workerraises adriver_healthwarning on its twin, whether or not the worker is running. -
One device per camera:
cyberwave edge installbinds each camera twin to its own camera and records the binding undertwin_to_devicein the camera config. For a USB camera the CLI pins a/dev/v4l/by-idlink, or a/dev/v4l/by-pathlink when two identical units are attached and a by-id name could match either; the value can also be an index (a camera that cannot be pinned),serial:<serial>(a depth camera that the driver resolves by serial) or a stream URL. On Linux, Edge Core passes it to the driver asCYBERWAVE_METADATA_VIDEO_DEVICE(_load_selected_camera_device: an index becomes/dev/video<N>, a link or URL goes through verbatim, aserial:value injects nothing), and a twin missing from the map gets the globalselected_deviceinstead. That fallback is why the installer records every pinned twin, even one whose camera the operator chose not to rebind, and every network camera with its stream URL. The injected/devnode is also watched, as described below.A by-id link resolves inside the container only if the container can see
/dev/v4l._ensure_linux_camera_docker_paramstherefore bind-mounts the host's/dev/v4lread-only into every image whose name containscamera-driver(orcamera_driver), and into any other image whose finalCYBERWAVE_METADATA_VIDEO_DEVICE, injected or set in its own params, lies under/dev/v4l/, unless the params already mount it.Three precedence rules decide what the driver actually receives.
_read_cameras_configprefersedge.json: a non-emptymetadata.camerasthere wins over the legacycameras.jsonfor every key and every twin it lists, and a twin listed only incameras.json'stwin_to_deviceis merged in, so it keeps that entry instead of falling back toselected_device. An explicit-e CYBERWAVE_METADATA_VIDEO_DEVICE=...in the driver's params wins over the mapping, and Edge Core then injects nothing. And the ALSA audio images, those whose name containsgeneric-microphone,microphone-driver,generic-speakerorspeaker-driver, never receive the variable at all. -
Network cameras: a camera on the Ethernet port (a FLIR AX8, an IP camera) has no
/devnode.cyberwave edge installfinds it on the wire, identifies it by MAC and writes its stream URL into the twin's metadata (video_device, plusmac_address) and intotwin_to_device, so Edge Core hands it to the camera driver like any other source; a URL has no node, so it is not watched for presence. When a driver param still names a source, that param wins as above, which is why the installer removes a placeholder or superseded param from the twin's drivers once it binds the device. How the device is found, matched and pinned, and how to describe a new kind of IP device in asset metadata, is in the CLI's wired network devices guide.
Use cyberwave edge status to see all driver and worker containers with their twin mappings.
Camera pins: drift and missing devices
Two reconcilers in the runtime loop (every ~15 s) keep camera drivers honest about the device they were given.
Pin drift. On Linux, reconcile_camera_config_drift notices a camera config change by a new modification time on edge.json or cameras.json. It then compares the CYBERWAVE_METADATA_VIDEO_DEVICE each running driver container was started with against the device its twin resolves to now: the twin's own twin_to_device entry, or the global selected_device when the twin has no entry or its entry resolves to no device (a serial: pin, for example). A container started without the variable is skipped, and so is a twin for which neither names a device. When the two differ, for example after cyberwave edge install --reconfigure-camera, Edge Core restarts the whole edge on a background thread through _perform_edge_core_restart: every driver, the worker and the Zenoh router are stopped and started again, not just the drifted driver. Microphone and speaker containers are never started with CYBERWAVE_METADATA_VIDEO_DEVICE, so they cannot trigger that restart, but it recreates them along with everything else.
Missing devices. reconcile_sensor_device_presence watches each twin that has a twin_to_device entry and a driver container started by this Edge Core, and it checks the device the launcher injects for that twin: the entry is resolved by _load_selected_camera_device, exactly as at launch, so an index becomes /dev/video<N>, a stable link is kept and a serial: pin resolves to nothing. Only a result under /dev/ is checked; serial pins and stream URLs have no node to look for. A node missing on two consecutive checks (about 30 s) raises one sensor_device_missing alert on the twin (error, source_type: "edge", metadata.device naming the path), and two consecutive checks that find it again resolve the alert, so a single glitch neither raises nor resolves one. The alert is also resolved as soon as the twin is no longer checked, because its entry was removed, it was re-pinned to a value with no /dev node, or no driver container is mapped to it any more. Camera drivers react to an unplugged device by reconnecting quietly, so without this check nothing told the operator why a camera had gone dark. Microphones are left out on purpose, because the CLI stores their pin in the twin's audio_device rather than in twin_to_device, and the microphone driver already raises its own sensor-failure alert while its pinned unit is absent. The "already alerted" state lives in memory, so after an Edge Core restart the first check that finds a twin's device present also resolves any alert left from before the restart, once per twin. A resolve that fails, for example while the backend is unreachable, is retried until it succeeds; an alert that fails to send is not retried.
Multi-container drivers
Some robots require multiple cooperating containers (e.g. a driver, bridge nodes, Nav2, SLAM, and elevation mapping). Edge Core supports this via an optional services array in the driver metadata. When present, Edge Core launches one container per service instead of a single driver container.
Metadata schema
{
"drivers": {
"linux-aarch64-jetson": {
"services": [
{
"image": "cyberwaveos/go2-ros2-driver:jetson-humble",
"name": "driver",
"command": ["ros2", "launch", "cyberwave_go2_driver", "robot_driver.launch.py"]
},
{
"image": "cyberwaveos/go2-ros2-driver:jetson-humble",
"name": "bridges",
"command": ["ros2", "launch", "cyberwave_go2_driver", "robot_bridges.launch.py"]
},
{
"image": "cyberwaveos/ros2-nav2:jetson-humble",
"name": "nav2"
},
{
"image": "cyberwaveos/ros2-slam:jetson-humble",
"name": "slam"
},
{
"image": "cyberwaveos/ros2-elevation-mapping:jetson-humble",
"name": "elevation",
"prefer_gpu": true
}
],
"shared_env": {
"CONFIG_PROFILE": "jetson",
"ROS_DOMAIN_ID": "0",
"CYBERWAVE_MAP_DIR": "/data"
},
"shared_params": ["--network", "host", "-v", "/data:/data"]
},
"default": {
"docker_image": "cyberwaveos/go2-ros2-driver",
"prefer_gpu": true
}
}
}
How it works
servicespresent → multi-container mode (one container per service entry).docker_imagepresent, noservices→ single-container mode (existing behavior, unchanged).- The
defaultfallback key works as before for platforms that don't match a specific key.
Per-service fields
| Field | Required | Description |
|---|---|---|
image |
Yes | Docker image reference |
name |
Yes | Service name (used in container naming) |
command |
No | Override the container entrypoint command |
env |
No | Per-service environment variables |
params |
No | Per-service Docker params |
prefer_gpu |
No | Enable GPU passthrough for this service |
gpu |
No | GPU device selector (default: all) |
Shared configuration
-
shared_env: Environment variables applied to every service. Per-serviceenvoverrides shared values. -
shared_params: Docker params applied to every service (e.g.--network host, volume mounts). -
service_image_overrides: for a driver whose image declares its own service graph (no inlineservices), maps service names to image references. Only those services' images change; the declared graph still supplies commands, env, networks, volumes and dependencies. Use it to try one service's build (a mapper, say) on a robot without restating the whole topology. A name the graph does not declare, or a value that is not a non-empty string, stops the driver from starting."service_image_overrides": { "slam": "cyberwaveos/ros2-slam:jetson-humble-<tag>" }
Environment layering
- Edge Core base env (
CYBERWAVE_API_KEY, MQTT, Zenoh, etc.) — existing, unchanged shared_envfrom metadata- Per-service
envfrom metadata
Container naming
- Single-container mode (unchanged):
cyberwave-driver-{twin_uuid[:8]} - Multi-container mode:
cyberwave-driver-{twin_uuid[:8]}-{service_name}
Backward compatibility
The existing single-image contract is fully preserved. Metadata without a services key follows the original code path with zero changes. All existing tests continue to pass unmodified.
Writing compatible drivers
A Cyberwave driver is a Docker image that interacts with device hardware and the Cyberwave backend. When Edge Core starts a driver container it sets the following environment variables (provided to the container):
CYBERWAVE_TWIN_UUIDCYBERWAVE_API_KEYCYBERWAVE_TWIN_JSON_FILE(writable file path; set when the twin file exists at launch, see Twin JSON file)CYBERWAVE_CHILD_TWIN_UUIDS(optional, comma-separated)CYBERWAVE_DATA_BACKEND— data transport backend (zenohby default)ZENOH_SHARED_MEMORY—true/false; enables zero-copy Zenoh SHM transportZENOH_CONNECT— (optional) comma-separated Zenoh router endpoint URLsCYBERWAVE_METADATA_VIDEO_DEVICE— (Linux, optional) the camera device this twin is pinned to, fromtwin_to_deviceor the globalselected_device. It is never set for microphone or speaker images, and it is left alone when the driver's own params already set it. When it names a/dev/v4l/link, the container also gets the host's/dev/v4lmounted read-only so the link resolves, whatever the image (see Multi-camera orchestration).
Any other -e CYBERWAVE_METADATA_<KEY>=... in a driver's params reaches the container as written and, by the convention the Cyberwave camera and microphone drivers follow, outranks the twin's metadata.<key>. That makes such a param the strongest way to fix a source, and also the easiest way to leave a stale one behind: a binding written to the twin's metadata is ignored while the param is there. Hardware found on the network is bound through the twin's metadata (the keys its asset's network_discovery.bind names, plus mac_address), so a driver for a new kind of IP device should read those keys from the twin JSON file below.
CYBERWAVE_CHILD_TWIN_UUIDS is present when child camera twins are attached to the driver twin; drivers can use this to coordinate cameras without additional prompts.
Zenoh data bus
Edge Core automatically injects Zenoh transport configuration into every driver container so that drivers using cw.data.publish() work without any extra configuration. The data-bus variables are:
| Variable | Default | Description |
|---|---|---|
CYBERWAVE_DATA_BACKEND |
zenoh |
Data transport: zenoh or filesystem |
ZENOH_SHARED_MEMORY |
false |
Opt-in zero-copy shared-memory transport. Requires --ipc=host between containers; leave disabled unless your runtime is configured for it. |
ZENOH_CONNECT |
(empty) | Router endpoints, e.g. tcp/10.0.0.1:7447 |
All variables can be overridden per-driver with -e KEY=VALUE in driver params.
Peer-to-peer mode (default): when ZENOH_CONNECT is empty, Zenoh uses multicast discovery. On Linux with --network host (the default), all driver containers on the same machine discover each other automatically.
Router mode (optional): set ZENOH_ROUTER_ENABLED=true to have Edge Core start an eclipse/zenoh:1.9.0 router container before the driver containers. This is required for MQTT bridge or multi-hop topologies.
Environment variables for Zenoh infrastructure:
| Variable | Default | Description |
|---|---|---|
ZENOH_ROUTER_ENABLED |
false |
Start a Zenoh router container before drivers |
ZENOH_ROUTER_IMAGE |
eclipse/zenoh:1.9.0 |
Docker image for the router |
ZENOH_ROUTER_PORT |
7447 |
Host port for the router |
ZENOH_SHARED_MEMORY |
false |
Opt-in shared-memory transport. Requires all Cyberwave containers to share an IPC namespace (--ipc=host); leave disabled unless validated end-to-end. |
Host telemetry
Edge Core reports the host's resource pressure over two transports. The bootstrap publisher folds the gauges into every cyberwave/twin/{uuid}/edge_health heartbeat it sends (~5 s), but Edge Core stops that publisher once driver startup has brought up at least one driver, because drivers publish their own health on the same topics from then on. After that the gauges travel only on the REST host-facts keepalive, which uploads them every ~30 s together with the static hardware facts. In practice, then, MQTT carries host metrics while the edge boots (and beyond that only on an edge whose startup brought up no driver), and REST carries them the rest of the time.
Dynamic (MQTT edge_health payload, every ~5 s while the bootstrap publisher runs):
| Field | Source | Notes |
|---|---|---|
host_memory_percent |
/proc/meminfo |
Absent on non-Linux hosts. From the monitor's 30 s snapshot. |
host_memory_available_mb |
/proc/meminfo |
Free + buffers/cache. From the monitor's 30 s snapshot. |
cpu_temp_c |
/sys/class/thermal/thermal_zone* |
Hottest CPU-typed zone. From the monitor's 30 s snapshot. |
host_cpu_percent |
/proc/stat |
Busy share of all cores between the monitor's last two snapshots. Absent on non-Linux hosts and before the second snapshot. |
host_disk_percent |
shutil.disk_usage |
Usage of the filesystem holding Docker's data root. From the monitor's 30 s snapshot. |
consecutive_critical |
SystemResourceMonitor |
Consecutive 30 s readings over a critical threshold (memory 92 %, CPU 82 °C). Resets on the first reading below both. |
power_mw |
sysfs power rails | Read fresh on every heartbeat; see Resource monitoring. |
battery_wh |
CYBERWAVE_BATTERY_WH |
Only when set. |
watchdog_layers |
ProcessWatchdog.active_layers() |
E.g. ["systemd"] or ["systemd", "hardware"]. Also on host_facts, which is where the dashboard reads it — see below. Still published here so a dashboard can keep showing layers for an edge whose binary predates that upload. |
Only the bootstrap publisher on the edge host injects these fields. Driver containers see their container's /proc, not the host's, so they intentionally omit the host block and publish only stream-level health.
Per-stream config:
Each entry under streams[id] may carry an optional stream_config block — a discriminated union keyed by kind (camera / audio / lidar / imu) describing the runtime-negotiated config for that stream. Drivers attach a block once at startup via EdgeHealthCheck.register_stream_config(stream_id, config) in the Python SDK; the publisher merges it into every heartbeat thereafter. The legacy top-level camera_config slot stays populated for one deprecation release for out-of-tree consumers and mirrors the lexicographically-first camera-kind stream_config. New consumers must read streams[id].stream_config instead — the legacy slot only ever describes one camera and is silent for audio / lidar / IMU streams.
reconcile_camera_config_drift works from file modification times and docker inspect output and never reads an edge_health payload, so changes to this schema cannot affect it (see Camera pins: drift and missing devices).
Static (REST POST /api/v1/edges/discover, every ~30 s):
Edge Core uploads a host_facts object — total RAM, CPU model and core count, kernel, thermal source, /dev/watchdog availability and the active watchdog layers, non-loopback network interfaces, a primary MAC address, the board serial number, and the current CPU-temperature / memory / power gauges (plus battery_wh when set) — into Edge.metadata['host_facts']. Those gauges duplicate the heartbeat fields above on purpose: the bootstrap publisher stops at the first driver start, and a host row that lost a reading there looked like a fault rather than a transport change. The first POST is performed by a background daemon thread as soon as it starts, then the same thread re-reads and re-POSTs every ~30 s (a fresh read each tick, not a cached payload — the interface list and the gauges below depend on that). Running the bootstrap POST off the main thread is deliberate: under systemd Type=notify, the service signals READY=1 from the main thread, and we don't want a slow or unreachable backend to push past TimeoutStartSec (which would cause systemctl restart cyberwave-edge-core to fail). The call is idempotent (it upserts Edge.metadata['host_facts'] and bumps Edge.last_seen_at), so the keepalive cost is one tiny round-trip per edge regardless of how many bound twins the edge has — or whether it has any at all.
network_interfaces — each interface's name, IPv4 address, MAC and up/down state — exists so an edge that reconnects to a different network (e.g. a different WiFi AP) can still be found and SSH'd into without scanning the LAN. Because it's the one field here that legitimately changes over the device's lifetime, Edge Core additionally publishes a one-line driver_log message ("Network interfaces: eth0=192.168.1.42 (up), ...") to every twin bound to the edge whenever the observed interface set changes, so the change shows up in that twin's Logs tab with a timestamp — not just as a silently-overwritten snapshot.
Three properties of that fan-out are load-bearing:
- Dedup is per twin, not per process. The set of bound twins grows over an edge's lifetime, so "already published" has to be a per-twin fact — a twin bound after the first publish has never seen the line and still gets one baseline message. Signatures live in memory only, so a restart re-publishes one baseline line per twin.
- A twin's signature is recorded only after its message reached a connected broker. The publish path swallows failures, so recording up-front would mean a broker that happened to be unreachable on the tick where the IP changed loses that line permanently. Nothing is recorded on failure, so every later tick retries until it lands.
- The fan-out runs on its own daemon thread, never on the keepalive thread. An MQTT connect against an unreachable broker can burn ~50 s (3 attempts × CONNACK timeout + backoff), which inline would stall the next
/discoverPOST and eat the margin onEdge.last_seen_at— a liveness signal whose whole point is to be independent of MQTT. Only one attempt is ever in flight; a tick that finds one running skips and retries later.
Interfaces with no IPv4 address bound are excluded from both the message and the change signature. They're nothing anyone can SSH to, and they churn (docker0/veth* as driver containers cycle, usb0 as cables move), which would otherwise re-fire the fan-out on changes that tell an operator nothing.
The Edges tab reaches the same goal by a different route: it renders only the interface named by primary_interface_name and keeps the full inventory on hover, so an unplugged NIC's MAC (what you pin a DHCP reservation to) and a VPN interface (how you reach a remote edge) stay one hover away without either crowding out the routable address. The exclusion here is specifically about log-line churn.
That Edge.last_seen_at field is what powers the three-state dashboard pill:
| State | Trigger | Meaning |
|---|---|---|
| Online | Edge.last_heartbeat fresh (≤60 s) |
A bound twin is publishing MQTT edge_health. |
| Standby | Edge.last_seen_at fresh (≤90 s) but no recent MQTT |
Edge Core daemon is alive and posting REST keepalives; no twin is doing work. |
| Offline | Neither signal within window | Process is down, or partitioned from both transports. |
The 30 s cadence is deliberately tighter than the 90 s Standby window so a single missed POST (slow DNS, transient TLS handshake) doesn't drop the row to Offline — three missed POSTs do. Failure of any individual upload logs at debug and falls through to the next tick.
Coverage per platform:
| Field | Linux | macOS | Notes |
|---|---|---|---|
platform, kernel |
yes | yes | From platform.platform() / platform.release(). |
memory_total_mb |
/proc/meminfo |
sysctl hw.memsize |
Rounded to one decimal on both platforms. |
cpu_model |
/proc/cpuinfo |
sysctl machdep.cpu.brand_string |
E.g. "Apple M2 Pro". |
cpu_count |
/proc/cpuinfo (processor: records) |
sysctl hw.logicalcpu → hw.ncpu |
Logical CPU count on both platforms (after SMT expansion). |
thermal_source |
first CPU thermal zone | omitted | Tracks the path the live publisher reads from. Darwin has no thermal reader today, so the field stays absent rather than claim a source that isn't sampled. |
has_hardware_watchdog |
/dev/watchdog |
always false |
Linux-only kernel capability. Says the device node exists, not that anything is pinging it — prefer watchdog_layers. |
watchdog_layers |
ProcessWatchdog.active_layers(), overlaid by Edge Core |
same | Which layers are actually supervising this process, in protocol order. The one key here the SDK's read_host_facts() does not produce: it probes /dev/watchdog and so cannot see systemd. Omitted, never [], when no layer is enabled, so "unprotected" and "edge predating this upload" stay distinguishable — the dashboard falls back to has_hardware_watchdog only for the second. Note that "hardware" is absent from the bootstrap POST even on a board that has the device: HardwareWatchdog.enabled only flips inside open(), which main.py calls after run_startup_checks() returns, so the layer appears one keepalive tick later. The dashboard renders that gap — and a permanently failing open() — as watchdog: systemd (+hw idle). Rides REST rather than the heartbeat for the same reason the gauges below do. |
network_interfaces |
/sys/class/net/* (name, address, operstate; IPv4 via SIOCGIFADDR) |
ifconfig -a (name, ether, status:, inet) |
The full inventory, deliberately unfiltered beyond loopback — a VPN interface (wg0, tailscale0) is often how you reach a remote edge, so this is not the place to decide what's "reachable". ipv4_address is null when the interface has no bound address (e.g. down); mac_address stays populated, since it's what a DHCP reservation keys on. On Linux, expect it to be long: an edge running driver containers reports 20+ entries (docker0, a br-* per compose network, a veth* per container, kernel tunnel pseudo-devices). Consumers should render primary_interface_name and keep the rest behind a disclosure. macOS narrows ~30 interfaces to the real NICs using three signals read out of the ifconfig text: a name prefix for the families Apple names consistently (utun*, awdl*, bridge*, anpi*, …); media: none, which means no physical layer at all and catches the internal anpi peers that are named en6–en9; and membership of a bridge*, which catches the Thunderbolt ports en2–en5 (the bridge stanza enumerates its own members, so their names are not assumed). An unplugged real NIC reports media: autoselect (none) — a media layer with no link — and is deliberately kept, since its MAC is what a DHCP reservation keys on. Measured on an M-series Mac: 32 interfaces → 3 (en0, en1, and one unplugged port), payload 587 B. Link state comes from status: rather than the RUNNING flag, which Apple's internal NICs assert permanently. |
primary_interface_name |
interface holding the default route | interface holding the default route | Name of the one interface the host actually routes out of, resolved by asking the kernel for the source address it would use (a UDP connect() to TEST-NET-1 — no packet is transmitted), then matching that address back to the enumerated set. Structural rather than heuristic: a bridge or veth peer cannot hold the default route, so no name blocklist is needed. Falls back to a hardware-backed interface (/sys/class/net/<if>/device exists) that is up when the host has no route at all. Omitted when the MAC came from the uuid.getnode() last resort. |
primary_mac_address |
MAC of primary_interface_name → uuid.getnode() |
MAC of primary_interface_name → uuid.getnode() |
The MAC — one value, resolved the same way on Linux, macOS, Jetson and Raspberry Pi, and the one to put in a DHCP reservation. Consumers need no selection logic of their own; picking client-side from network_interfaces by enumeration order lands on docker0 (sysfs enumeration is alphabetical and container bridges are up). Falls back to uuid.getnode() when nothing was enumerated — not tied to a reachable interface, but the only MAC available there; primary_interface_name is then omitted so the two provenances stay distinguishable. Omitted entirely when getnode() fabricated a random node ID (RFC 4122 flags this with the multicast bit), since a MAC-shaped string matching no NIC is worse than nothing. |
device_serial |
Serial in /proc/cpuinfo (Raspberry Pi) → /sys/firmware/devicetree/base/serial-number (Jetson) |
omitted | Board serial for asset tracking / RMA. Device-tree values are NUL-terminated and stripped. All-zero placeholder serials (0000000000000000) are treated as absent. |
cpu_temp_c |
hottest CPU thermal zone | omitted | Live gauge, not a fact — see below. Deliberately the hottest zone, while thermal_source names the first; the two disagree by design. |
memory_used_percent, memory_available_mb |
/proc/meminfo |
omitted | Live gauges — see below. sysctl gives Darwin the total but not the free/available split, so both stay absent there. |
sdk_version, edge_core_version |
in-process __version__ → importlib.metadata |
in-process __version__ → importlib.metadata |
Version of the SDK and Edge Core actually loaded by the running daemon. The in-process lookup preserves the BUILD_VERSION stamp injected by CI for .deb builds and pre-release wheels, and survives PyInstaller binaries that ship without .dist-info, so the Edges tab agrees with cyberwave-edge-core --version. CLI version is intentionally not part of host_facts — the CLI ships as a separate PyInstaller binary, so Edge Core cannot observe it from its own Python process; cyberwave --version remains the source of truth for the CLI an operator is running locally. |
Why two gauges ride a facts payload
cpu_temp_c and the memory_* gauges are the same readings the MQTT edge_health heartbeat publishes every ~5 s, duplicated onto the ~30 s REST keepalive. The duplication is not redundancy for its own sake — without it those readings have no route off a working edge at all:
- The bootstrap
edge_healthpublisher is the only thing that samples them, and Edge Core stops it as soon as the first driver starts (drivers own the twin's health topic from then on). - A containerised driver sees its own
/procand has no access to the host's thermal sysfs, so it cannot take over the sampling.
The net effect before this existed: memory and temperature appeared while an edge sat idle and vanished the moment it started doing work — exactly backwards from when an operator wants them. Consumers that may see both copies should prefer whichever is fresher, which is the heartbeat's only while heartbeats are still arriving — a consumer holding the last-received payload in memory keeps those numbers indefinitely, so once the heartbeat stops they become a frozen snapshot and this copy is the live one. The dashboard treats a stale heartbeat as having no gauges at all and falls through to these. Treat a fallback reading as up to 30 s stale at the source, plus whatever polling interval the consumer adds on top.
Local pages and the edge local portal (opt-in)
A driver that serves a page of its own on the edge's network (a download page
for a ground-station browser, a camera preview, a status page) declares it in
its cw-driver.yml:
local_http: # shorthand for drivers.default.local_http
- name: plans # slug, unique per driver
port: 8765 # the port the driver itself listens on
path: / # absolute path; no query string or fragment
service: gateway # optional: which service of a multi-container driver serves it
audience: lan # lan (default) is listed; host is declared but never listed
scheme: http # http (default) or https
description: Plans for the ground station
The backend validates the block and stores it on the driver profile
(drivers.<profile>.local_http); a twin can override it through its own
metadata.drivers, with the same precedence Edge Core uses to pick the image.
The driver still opens its own port: the declaration only makes it
discoverable.
With the portal switched on, Edge Core serves a small read-only index of those pages:
| Variable | Default | Meaning |
|---|---|---|
CYBERWAVE_EDGE_LOCAL_PORTAL |
off | on / true / 1 starts the portal. Anything else leaves Edge Core with no HTTP surface. |
CYBERWAVE_EDGE_LOCAL_PORTAL_BIND |
0.0.0.0:8700 |
host:port, [v6]:port, :port or port. Use 127.0.0.1:8700 to keep it on the host. |
CYBERWAVE_EDGE_LOCAL_PORTAL_HOST |
address the client reached | Host name or IP printed in links, e.g. a DNS name your LAN resolves. |
CYBERWAVE_EDGE_LOCAL_PORTAL_REFRESH_SECONDS |
10 |
How often the listing is rebuilt from docker ps and the twin files. |
Set them in the Edge Core service environment or under envs in
credentials.json, then restart Edge Core.
GET /lists, per running driver, its twin name, registry id, health (from the container healthcheck) and a link tohttp://<edge-ip>:<port><path>for each declared page. A page whoseservicecontainer is not running is shown asstopped, without a link.GET /portal.jsonis the same listing, machine-readable (schema: cyberwave.edge.local_portal.v1).GET /healthzreports the portal's own liveness.
What it does not do, by design:
- No proxying. Links point straight at the driver's port, so the portal adds an index and nothing else to the edge's attack surface. Protecting a page (if it needs it) is the driver's job.
- No secrets. The output is built from a fixed set of fields. Tokens,
shared_env, image names and the rest of the twin metadata never reach it, and declared paths cannot carry a query string. Entries are re-validated on the edge, because twin metadata is editable after the catalog checked it. - No mDNS. Edge Core has no mDNS dependency and this does not add one; reach the portal by IP or by a name your network already resolves.
- Never blocks the runtime loop. Docker and the twin files are read by a background refresher; requests are answered from that snapshot. A bad bind value or a busy port is logged and Edge Core runs on without the portal.
The links use the address the client connected to (or
CYBERWAVE_EDGE_LOCAL_PORTAL_HOST), never the request's Host header.
Driver failure handling
Drivers should exit with a non-zero code when they cannot access required hardware at startup (for example, missing /dev/video* or disconnected peripherals). This allows Edge Core to detect startup failures and trigger restart logic. Not every driver does: the camera driver keeps running and reconnects when its device is unplugged, which is why Edge Core watches camera pins itself (sensor_device_missing below).
Edge Core raises these twin alerts, all with source_type: "edge":
| Alert type | Severity | Raised when | Cleared by Edge Core |
|---|---|---|---|
driver_starting |
info |
A driver start attempt begins; carries pull and startup progress (see Remote restart). | Resolved once the container runs. A failed attempt is annotated with its phase, escalated to critical and resolved. |
worker_starting |
info |
The same for the ML worker, held until the worker reports ready (see Worker readiness). | As above. |
driver_start_failure |
error |
The twin has no asset, the driver image cannot be pulled, or the container does not reach a running state. A launch refused only by its restart floor raises nothing. | Resolved by the next successful start of that twin's driver. |
driver_restart_loop |
error |
The driver restarted more than CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD times within CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS. The container is switched to --restart=no and stopped. |
No. |
driver_health |
warning |
A driver container that was running is found stopped or removed. | No. |
sensor_device_missing |
error |
The /dev node a managed camera twin is pinned to is missing on two consecutive checks (see Camera pins: drift and missing devices). |
Resolved when two consecutive checks find it again, or when the twin is no longer checked. |
driver_restart_needed |
warning |
A configuration key in the metadata of a twin whose driver is not a sensor driver, or declares a restart floor, changed (see Bidirectional twin sync). Edge Core restarts nothing. | No. |
driver_metadata_restart_suppressed |
warning |
A sensor driver has already been restarted three times in 30 minutes for metadata changes; the change is left for a manual restart. | No. |
edge_cpu_saturation, edge_memory_saturation, edge_disk_saturation |
warning |
Raised on every managed twin when CPU, memory or disk usage is at or above CYBERWAVE_RESOURCE_SATURATION_PERCENT on two consecutive 30 s snapshots (see Resource monitoring). |
Resolved after two consecutive snapshots five points below the threshold. |
model_download_failure |
error |
A required ML model cannot be downloaded. | No. |
worker_start_failure |
error |
The ML worker container cannot be started, or its startup probe gives up (see Worker startup probe). | Resolved by the next successful worker start. |
workflow_sync_failure |
error |
Downloading a twin's workflow workers from the backend failed. | No. |
macos_camera_not_configured, macos_microphone_not_configured, macos_speaker_not_configured |
warning |
On macOS, a camera, microphone or speaker twin has no host stream or bridge URL configured. | No. |
When a model fails to download, the worker container is not started: a workflow that depends on the model is held off rather than allowed to run with empty inference results. Models that the worker container can resolve on its own (an Ultralytics auto-download on a frozen PyInstaller binary, for example) are deferred via _RuntimeManagedDeferred and never trip the alert.
docker_create_timeout is a failure phase, not an alert type. When docker create times out and the container has still not appeared 10 s later (see When docker create times out), the driver_starting alert is closed with phase: "docker_create_timeout" and the twin gets a driver_start_failure. Nor does Edge Core create the edge_core_restart alert: the backend creates it when a restart is requested through POST /api/v1/edges/{uuid}/restart-core and ships its UUID in the MQTT command, and Edge Core only moves its metadata.phase from in_progress to completed or failed (EdgeCoreRestartAlertContext in utils.py).
Optional environment variables to tune restart behavior:
CYBERWAVE_DRIVER_RESTART_LOOP_THRESHOLD(default:4)CYBERWAVE_DRIVER_RESTART_LOOP_WINDOW_SECONDS(default:60)CYBERWAVE_DRIVER_CREATE_TIMEOUT_SECONDS(default:30)CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS(default:300;0keeps removed drivers down)CYBERWAVE_DRIVER_TROUBLESHOOTING_URL(default:https://docs.cyberwave.com)
Periodic Docker cleanup
Edge Core prunes stopped cyberwave-* containers every 30 minutes, and unused images and anonymous volumes every 3 hours, from the runtime loop (_run_periodic_docker_cleanup). When Docker's data root is on an SD card the pruning is deferred to spare the flash, but only until that filesystem is 80 % full; CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP=1 turns it off altogether. The intervals, the threshold and the reasoning are under Disk usage: container logs and cleanup.
Driver revival and orphan containers
When a managed driver exits cleanly (Docker's --restart unless-stopped policy does not auto-revive clean exits), Edge Core's revival reconciler re-runs driver startup for that container so it is recreated. Revival is restricted to driver containers this Edge Core process launched. A service declared restart: "no", or one the flap detector stopped, is left down, unless its relaunch was refused only on its restart floor. If docker ps fails or times out, the health snapshot from the previous cycle is kept and nothing is classified, alerted or revived on the strength of the failed listing.
A managed container that has been removed altogether, by docker rm or by the periodic prune, is revived too, but only after a grace period: 300 s by default, set with CYBERWAVE_DRIVER_REMOVED_REVIVAL_SECONDS, where 0 disables it. The grace leaves an operator time for a deliberate takedown, an image swap or a debugging session before Edge Core brings the driver back. _removed_containers_due notes when each container was first seen removed and hands the overdue ones to the same revival pass as clean exits, so the orphan filter, the flap detector's opt-out and the 60 s revival debounce all apply. A removed container cannot be inspected, so Edge Core records the restart policy it launched each container with, and marks the ones the flap detector stopped as no: a removed restart: "no" or flap-stopped driver is not revived, unless its relaunch was refused on its restart floor. If a revival pass finds no linked twin that still declares a removed container, the container is not forgotten at once: that takes three passes in a row without a declaring twin (_REMOVED_FORGET_AFTER_MISSES), each at least a minute apart, so a twin list that comes back short while the backend is flaky does not cost a driver, and a twin that is really gone costs three passes rather than one a minute. Containers removed by an edge restart are dropped from Edge Core's bookkeeping and never come back this way; the restart relaunches what it still needs.
Stopped cyberwave-driver-* containers that this process did not start, such as those of twins unlinked before Edge Core last started, are treated as orphans and ignored by revival. The periodic prune removes them with every other stopped cyberwave-* container, except those that carry a restart-floor label, or when periodic cleanup is disabled. Without this guard, an orphan would send a revival pass to the backend every minute for a container that no linked twin declares.
Twin JSON file
CYBERWAVE_TWIN_JSON_FILE points the driver at /app/<twin_uuid>.json, a JSON copy of its twin: the twin instance as the API returns it (TwinSchema, including metadata), with the catalog asset (AssetSchema) under asset. Edge Core sets the variable only when the host file exists at launch.
On the host the file is <config_dir>/<twin_uuid>.json, and the container sees it twice: as a writable single-file bind mount at /app/<twin_uuid>.json, and inside the read-only mount of the whole config directory at /app/.cyberwave/<twin_uuid>.json. The difference matters because a single-file bind mount follows the file's inode, not its path. That is why write_or_update_twin_json_file, which runs before every driver launch, deep-merges the backend twin into an existing file and rewrites it in place rather than replacing it: the inode a running container holds stays the one the host sees.
Drivers read the file when they start. The Cyberwave camera driver's entrypoint.sh, for example, exports the twin's metadata as CYBERWAVE_METADATA_* variables once, before the driver process runs, so a change to the file while the driver runs does not reach the driver until it is started again.
Bidirectional twin sync
reconcile_twin_json_file_sync runs on every runtime tick (~15 s) and keeps the host files and the backend in step in both directions.
The push leg looks for files whose SHA-256 changed since the previous tick and sends each one with PUT /api/v1/twins/{uuid}. The payload is every field of _TWIN_UPDATE_ALLOWED_FIELDS present in the file: the whole metadata object along with the name, pose, attachment and calibration fields, but never asset or environment assignments. This is how a driver's own edits to its twin file reach the backend. A file seen for the first time only has its checksum recorded, and the writes Edge Core makes itself, at launch and on a pull, record the new checksum too, so they are never pushed back.
The pull leg covers every file the push leg did not send in that tick. _pull_twin_json_file_from_backend fetches the twin with client.twins.get_raw and, for each field in _TWIN_PULL_ALLOWED_FIELDS (only metadata), compares the backend value with the local one as a whole; if they differ at all, the backend value replaces the local one outright, with no key-by-key merge. A push that failed therefore does not protect a local edit, because the same tick's pull can overwrite it with the backend's metadata. For the same reason a field the edge legitimately writes must never be added to _TWIN_PULL_ALLOWED_FIELDS.
Like the launch path, the pull rewrites the host file in place, so the inode a running container holds stays the one the host sees: /app/<twin_uuid>.json shows the pulled content at once, and what the driver writes there still reaches the push leg. The new content is first written to a temporary file, so a full disk fails before the original is touched, and the original is then overwritten without being emptied first, a shorter document padded with spaces until the old tail is cut, so a reader always finds valid JSON. A host file that cannot be read is rebuilt from the last copy Edge Core wrote or read intact, with the backend metadata applied; with no such copy in memory it is left, with one warning, for the next launch to rewrite. The pull reports which top-level metadata keys changed, and reconcile_twin_json_file_sync returns them per twin.
Drivers read the file only when they start, so a metadata edit made in the UI, such as a new microphone gain, reaches a driver only through a restart. reconcile_driver_metadata_restarts, which runs right after the sync, makes that restart for sensor drivers: the camera, microphone, speaker and webcam drivers, recognised by their image names.
- Configuration only. Runtime state is left out of the comparison: telemetry (
status,power,motor_temps,performance), the backend's navigation keys (active_map_*,finalize_active_map_request,last_finalized_map_*,navigation_lifecycle,localization), the drivers' own reports (speaker_device,audio_routing,robot_dump, andaudio_devicewhen it is the webcam's dict rather than a pinned device string) andedge_fingerprint; a change ofedge_fingerprint, a link or unlink, never restarts anything. Insensors, entries withoutparametersare left out too, because a driver registers a missing sensor on its own as a bare id and type, and a list left empty counts as nosensorsat all. Every other key is configuration, compared as a digest of the filtered metadata. - Debounced. The restart happens once the filtered metadata has been unchanged for at least 10 s, that is on a later tick, so several saves in a row cost one restart.
- The driver's own writes do not count. The digest written at launch is the baseline, and whatever Edge Core pushes from the file becomes the new one; the writes a driver is known to make through the backend are filtered out of the digest (see above). Each of the twin's containers is judged on its own: one that started after the change, or is not running, has it or reads it when it starts. A change seen during a container's first 90 s waits for those 90 s to pass, so nothing restarts while a driver is starting; if the filtered metadata still differs from what the containers started with, the twin is then restarted. A real edit made right after a restart therefore still reaches the driver, and a startup write Edge Core does not recognise costs at most the restarts the rate limit allows. A container that cannot be inspected holds the decision until a later tick.
- Relaunch. The twin's containers go through
fetch_and_run_twin_drivers(..., only_container_names=..., pull_images=False), the revival path, which re-fetches the twin, rewrites the file and showsdriver_starting. A child camera twin restarts the containers of the twin it is attached to. The relaunch uses the image already on disk and pulls only a missing one: a settings change applies settings and does not upgrade the driver. It also runs inside the runtime loop, whose onlyWATCHDOG=1comes at the end of each cycle, so a pull of a newly published tag could outlastWatchdogSec=60and get Edge Core aborted. For the same reason the relaunch gets the loop's watchdog, which the dependency health wait pings. The change counts as applied only when every relaunched container started; otherwise it stays pending and is retried under the rate limit below. - Rate limit. At least 120 s between two metadata restarts of a twin and at most three in 30 minutes; beyond that the change is left for a manual restart and the twin gets one
driver_metadata_restart_suppressedalert. - Never during another restart. The step waits while an Edge Core or camera-drift restart is in progress.
Any other driver, a robot or a drone, and any service with a restart floor, is not restarted for a metadata change: the change raises one driver_restart_needed alert ("Settings changed; restart the driver to apply them") on the twin whose settings changed. Until that alert, or a driver_metadata_restart_suppressed one, reaches the backend, the change stays pending and the alert is retried on later ticks.
Twin metadata
Use the official Cyberwave SDK to interact with the API and MQTT; it abstracts authentication, retries, and handshake logic.
Register a driver by adding its configuration to a twin's metadata (or the catalog twin's metadata if you control the catalog twin). Use the environment view's Advanced editing to edit metadata.
Note: changing a catalog twin's metadata affects all subsequently created digital twins derived from that catalog twin.
Example driver metadata (JSON):
{
"drivers": {
"default": {
"docker_image": "cyberwaveos/so101-driver",
"version": "0.0.1",
"params": [
"--network",
"local",
"--add-host",
"host.docker.internal:host-gateway"
]
}
}
}
Platform-specific driver selection
Edge Core can select platform-specific driver entries before falling back to
default.
Selection order:
- Child-registry-specific entry (existing behavior)
- Host platform/machine keys (for example
darwin-arm64,darwin,macos,mac) default
Example:
{
"drivers": {
"default": {
"docker_image": "cyberwaveos/so101-driver"
},
"darwin-arm64": {
"docker_image": "cyberwaveos/so101-driver:macos",
"params": ["-e", "CYBERWAVE_SERIAL_BRIDGE_URL=tcp://host.docker.internal:22001"]
}
}
}
macOS host-device bridge hook
On macOS, Linux --device mappings in params cannot directly expose host
hardware to Linux containers. Edge Core now supports a pre-run native bridge
hook:
- Set
CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMANDon the host - Edge Core executes it once per
--devicemapping beforedocker run - Template variables available:
{host_device}{container_device}{twin_uuid}{container_name}{config_dir}
Example:
export CYBERWAVE_MACOS_DEVICE_BRIDGE_COMMAND="cyberwave-edge-hw-bridge --device {host_device} --target {container_device} --twin {twin_uuid}"
The command can start native camera/serial forwarding services that expose
bridge endpoints to the container (typically via host.docker.internal).
Bridge command stdout can optionally return a resolved source for the mapped device:
- JSON:
{"resolved_device":"rtsp://host.docker.internal:8554/cam0"} - or line format:
resolved_device=rtsp://host.docker.internal:8554/cam0
When this value differs from /dev/video*, Edge Core can transparently:
- inject
CYBERWAVE_METADATA_VIDEO_DEVICEfor the driver - inject
CYBERWAVE_CHILD_CAMERA_STREAM_URLSwith the per-child MJPEG mapping for multi-camera parent drivers such as SO101 - inject
CYBERWAVE_EDGE_VIDEO_DEVICE_MAP(JSON map of Linux device to resolved source) - remove Linux-only
--device /dev/video*flags beforedocker runon macOS (default enabled withCYBERWAVE_MACOS_STRIP_VIDEO_DEVICE_PARAMS=true)
This lets Linux-style drivers keep their normal auto-setup logic while receiving a macOS-compatible video source without driver code changes.
For camera twins, Edge Core can also provide default bridge candidates on macOS
even when metadata has no explicit --device params (default driver config),
so Linux-oriented camera drivers remain compatible with minimal metadata.
To inject environment variables into a driver container, list -e flags inside params: each -e must be a separate element followed by its KEY=value string. Example:
{
"drivers": {
"default": {
"docker_image": "cyberwaveos/go2-native-driver",
"params": ["-e", "MY_VAR=value", "-e", "ANOTHER_VAR=value2"]
}
}
}
Each -e must be its own element in the array, followed by the KEY=value string as the next element. This is equivalent to passing -e MY_VAR=value on the docker run command line.
This is useful for driver-specific configuration that varies per device, such as IP addresses, credentials, or feature flags that cannot be stored in the twin's edge_configs metadata.
Runtime configuration for drivers (metadata["edge_configs"])
Drivers and edge services should treat metadata["edge_configs"] as the source of truth for per-device runtime configuration.
Edge identity should be stored at metadata["edge_fingerprint"] (not duplicated inside edge_configs).
Runtime access: The core passes the full twin JSON (including
metadata) to every driver via theCYBERWAVE_TWIN_JSON_FILEenvironment variable. Drivers can readedge_configsfrom that file at startup to obtain per-device settings — for example, selecting the right camera source or IP address for the current machine. This is the recommended way to pass device-specific configuration to a driver without hardcoding values in the image.
- Type: object/dictionary
- Value: binding object (
object)
Canonical shape:
{
"edge_fingerprint": "macbook-pro-a1b2c3d4e5f6",
"edge_configs": {
"camera_config": {
"camera_id": "front",
"source": "rtsp://user:pass@192.168.1.20/stream",
"fps": 10,
"resolution": "VGA",
"camera_type": "cv2"
}
}
}
Field notes:
edge_fingerprint: fingerprint of the edge serving this twin (recommended).camera_config: per-device camera/runtime config consumed by drivers.
Avoid storing transient runtime state such as edge_uuid, registered_at, last_sync, last_ip_address, or status_data inside edge_configs.
Backward compatibility:
- Older records may use a legacy map shape (
edge_configs[fingerprint] = {...}). - Older records may store camera settings in
cameras[0]or as top-level fields. - New writers should prefer
camera_configunderedge_configs. - Do not rely on
PUT /api/v1/edges/{uuid}/twins/{twin_uuid}/camera-config; it is deprecated. Update twin metadata instead.
Advanced usage
Manual install and troubleshooting
# Install the Buildkite package signing key
curl -fsSL "https://packages.buildkite.com/cyberwave/cyberwave-edge-core/gpgkey" | gpg --dearmor -o /etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg
# Configure the Apt source
echo -e "deb [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/cyberwave_cyberwave-edge-core-archive-keyring.gpg] https://packages.buildkite.com/cyberwave/cyberwave-edge-core/any/ any main" \
> /etc/apt/sources.list.d/buildkite-cyberwave-cyberwave-edge-core.list
# Run Edge Core (performs startup checks and starts drivers + worker container)
cyberwave-edge-core
# Show status, credentials and MQTT connectivity (read-only)
cyberwave-edge-core status
# Show version
cyberwave-edge-core --version
# Worker container management (also available via `cyberwave worker …`)
cyberwave-edge-core worker start # Start the worker container
cyberwave-edge-core worker stop # Stop the worker container
cyberwave-edge-core worker restart # Restart the worker container
cyberwave-edge-core worker status # Show container state, workers, cached models, and health
cyberwave-edge-core worker health # Show detailed restart history and circuit-breaker state
cyberwave-edge-core worker logs # Stream worker container logs
cyberwave-edge-core worker logs --no-follow # Print recent logs without following
Preview builds from dev / staging CI are published as separate Debian packages in the same apt repo: cyberwave-edge-core-dev and cyberwave-edge-core-staging. apt install cyberwave-edge-core only pulls tagged releases; use one of the channel packages explicitly when you want those binaries (the packages conflict because they ship the same /usr/bin/cyberwave-edge-core).
On non-apt platforms, prerelease Python wheels are published to the Buildkite Python registry and consumed automatically by cyberwave edge install --channel dev|staging. Stable pip installs continue to use the public PyPI release.
Environment variables
Run against a different environment/base URL:
export CYBERWAVE_ENVIRONMENT="yourenv"
export CYBERWAVE_BASE_URL="https://yourbaseurl"
cyberwave-edge-core
Control log verbosity (default: INFO). Edge-core forwards any
CYBERWAVE_* env var it sees in its own process environment into each
worker container at docker run time, so the same knobs work whether
they're set in the systemd unit, the CLI install env, or the shell:
# Edge-core + edge drivers (camera/UGV/etc.)
export CYBERWAVE_EDGE_LOG_LEVEL="DEBUG"
# Worker containers only — wins over CYBERWAVE_EDGE_LOG_LEVEL when set,
# so you can keep workers at INFO while a driver is at DEBUG (or vice versa).
export CYBERWAVE_WORKER_LOG_LEVEL="INFO"
cyberwave-edge-core
Or pass env vars to the CLI installer:
sudo CYBERWAVE_ENVIRONMENT="yourenv" CYBERWAVE_BASE_URL="https://yourbaseurl" CYBERWAVE_MQTT_HOST="yourmqtt" cyberwave edge install
Disk usage: container logs and cleanup
Docker's default json-file log driver has no size limit, and every container
Edge Core launches runs under --restart unless-stopped. On a device with a
small disk — a Raspberry Pi on a 16 GB uSD, say — a chatty driver can fill the
card in days. Edge Core therefore caps each container's log at launch rather
than relying on the host's /etc/docker/daemon.json.
| Variable | Default | Purpose |
|---|---|---|
CYBERWAVE_DOCKER_LOG_MAX_SIZE |
10m |
Max size per container log file before rotation. Set to off to emit no log flags at all and inherit the host daemon's configuration — use this if you ship logs through a daemon-wide driver such as journald or fluentd. |
CYBERWAVE_DOCKER_LOG_MAX_FILE |
3 |
Rotated files kept per container (so 30 MB per container by default). |
CYBERWAVE_DRIVER_LOG_COLD_START_TAIL |
500 |
Backlog lines replayed when Edge Core starts following an already-running container's logs. 0 restores the previous unbounded replay. |
Per-container flags override daemon.json, so the defaults match what the Pi
image already sets daemon-wide — installing Edge Core never raises a cap the
host had already chosen. Raise them explicitly if you want more history.
A driver's catalog docker_run_params win, at the granularity they were set:
an explicit --log-driver means Edge Core adds nothing at all, while an
explicit --log-opt max-size (with no driver) keeps that value and still gets
the default max-file bound.
Periodic cleanup prunes stopped Cyberwave containers (every 30 min) plus unused images and anonymous volumes (every 3 h):
| Variable | Default | Purpose |
|---|---|---|
CYBERWAVE_SKIP_PERIODIC_DOCKER_CLEANUP |
unset | Set to 1 to disable all periodic pruning. |
CYBERWAVE_SD_CARD_CLEANUP_USAGE_PERCENT |
80 |
When Docker's data root is on an SD card, routine pruning is deferred to limit flash wear — but runs anyway once that filesystem reaches this percentage full. |
CYBERWAVE_CONTAINER_PRUNE_INTERVAL_SECONDS |
1800 |
Stopped-container prune interval. |
CYBERWAVE_IMAGE_PRUNE_INTERVAL_SECONDS |
10800 |
Unused-image prune interval. |
CYBERWAVE_VOLUME_PRUNE_INTERVAL_SECONDS |
10800 |
Anonymous-volume prune interval. Named volumes are never pruned. |
Volume pruning requires Docker 23.0 or newer, where docker volume prune
removes only anonymous volumes. On older daemons the same command would also
delete unused named volumes belonging to other workloads on the host, so Edge
Core skips it and prunes containers and images only. The docker-ce packages
our installer and Pi image use are well past 23.0; the distro docker.io
package is not.
If a device is running out of space, start here:
docker system df -v
sudo ls -lSh /var/lib/docker/containers/*/*-json.log | head
du -sh ~/.cyberwave/models/
journalctl --disk-usage
Local development (from this folder)
You can develop both the Cyberwave CLI and Edge Core from the cyberwave-edge-core directory using a single virtual environment that has the monorepo SDK, CLI, and edge-core installed in editable mode.
One-time setup
From cyberwave-edge-core/:
# Create and activate a venv (e.g. .venv in this folder)
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install SDK, CLI, and Edge Core in editable mode (order matters: SDK first)
pip install -e ../cyberwave-sdks/cyberwave-python
pip install -e ../cyberwave-clis/cyberwave-python-cli/"[build]"
pip install -e ".[build]"
Generate the SDK REST client (required for editable SDK). The SDK’s cyberwave.rest package is generated from the backend OpenAPI spec and is not committed. If you see ImportError: cannot import name 'DefaultApi' from 'cyberwave.rest':
- Start the backend:
cd ../cyberwave-backend && docker compose -f local.yml up -d(wait until healthy). - From the repo root, generate the REST client:
cd cyberwave-sdks && ./python-sdk-gen.sh sdk --host localhost:8000
- Re-run the
pip install -esteps above if you already installed; the editable SDK will then include the generatedcyberwave/restcode.
Run CLI and Edge Core
After activating the venv, both commands are on your PATH:
# CLI
cyberwave --help
cyberwave login --email boss@cyberwave.com --password iamnottheboss
cyberwave edge install --help
# Edge Core
cyberwave-edge-core --help
cyberwave-edge-core status
cyberwave-edge-core
Target backend: If you do not set CYBERWAVE_BASE_URL, the CLI and Edge Core use the default production API (https://api.cyberwave.com). To use your local backend instead:
export CYBERWAVE_BASE_URL=http://localhost:8000
export CYBERWAVE_MQTT_HOST=localhost
export CYBERWAVE_ENVIRONMENT=local
Paths from this folder
| What | Path (from cyberwave-edge-core/) |
|---|---|
| Repo root | .. |
| Python SDK | ../cyberwave-sdks/cyberwave-python |
| CLI | ../cyberwave-clis/cyberwave-python-cli |
Edit code in any of those directories; the editable installs pick up changes (no reinstall needed for Python changes).
Testing
CI (.github/workflows/edge-core-test-and-push.yml) runs the whole unit suite, pytest tests/ --ignore=tests/integration, and then the Docker integration tests separately. Run the same thing locally from this folder:
pytest tests/ --ignore=tests/integration -q
The systemd behaviour (readiness, the watchdog, bounded start and stop, upgrades of the Debian package) can only be checked under a real systemd. scripts/test_deb_upgrade_under_systemd.sh boots one in a container, installs the package given by PACKAGE_FILE and checks install, upgrade, hold and update. It then runs the lifecycle test files of the CLI and of Edge Core inside that systemd, with CYBERWAVE_TEST_SYSTEMD_SYSTEM=1 and CYBERWAVE_REQUIRE_SYSTEMD_TESTS=1 so that tests needing systemd fail instead of skipping. The Edge Core checkout is mounted read-only. LIFECYCLE_ONLY=1 skips the Debian checks and needs no package, which is the quick way to check a lifecycle change:
LIFECYCLE_ONLY=1 bash scripts/test_deb_upgrade_under_systemd.sh
PACKAGE_FILE=path/to/package.deb bash scripts/test_deb_upgrade_under_systemd.sh
Contributing
Contributions are welcome. Please open an issue to discuss bugs or feature requests, and submit a pull request when you are ready.
Community and Documentation
- Documentation: https://docs.cyberwave.com
- Community (Discord): https://discord.gg/dfGhNrawyF
- Issues: https://github.com/cyberwave-os/cyberwave-edge-core/issues
Metadata
Release files for cyberwave-edge-core 0.1.39
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| cyberwave_edge_core-0.1.39.tar.gz | 746.7 kB | With provenance | Details for cyberwave_edge_core-0.1.39.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| cyberwave_edge_core-0.1.39-py3-none-any.whl | Python 3 | none | any | Details for cyberwave_edge_core-0.1.39-py3-none-any.whl |
Total release size: 1.1 MB
Release files / cyberwave_edge_core-0.1.39.tar.gz
| Download URL | cyberwave_edge_core-0.1.39.tar.gz |
|---|---|
| Size | 746.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9ca3052926da6d5a7d7a7143c633760644faa5bad3c2c093d47ded797820e3fb
|
|
BLAKE2b-256 checksum How to use checksums |
e40ac9ee17ce6d4de0adc4abd8d52ab7563d4977118bdfde6153c66b89113382
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logRelease files / cyberwave_edge_core-0.1.39-py3-none-any.whl
| Download URL | cyberwave_edge_core-0.1.39-py3-none-any.whl |
|---|---|
| Size | 390.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f90b59a95b9f3af325ce77aabe947f31f1b987a8882a747edfe9633d4b6bb6df
|
|
BLAKE2b-256 checksum How to use checksums |
2c60e26f19c3e662e32249e677c31999d58394f176121ac5938fad208c653139
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logSecurity for cyberwave-edge-core 0.1.39
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for cyberwave_edge_core-0.1.39.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.