Toposync Vision extension

First-party extension focused on public task-oriented vision operators for the Pipelines runtime.

What it provides

• vision.detect
• vision.track
• vision.group_events
• vision.spatial_relation_event confirms proximity after observed dwell. A valid matched pair stays stable when another member moves closer. Replacing either participant closes the previous occurrence and starts a new dwell; time without a new observation cannot confirm a candidate. These events describe proximity, not intent or recognition across separate visits.
• vision.identity_evidence and vision.recognize_identity (explicit activation; setup, curation and recovery)
• vision.crop_objects
• vision.segment_instances
• vision.pose_estimate (experimental MediaPipe Pose 33 and RTMPose-M Halpe26 backends)
• vision.gesture_recognize (experimental temporal body gestures)

The public surface is task-based, not vendor-based. The official first-party runtime is ONNX Runtime, with CPU as the default execution path.

For spatial relations, dwell_seconds accumulates intervals between consecutive positive observations. Loss of evidence pauses this duration; recovery preserves the occurrence within its grace period without adding the missing interval. Closing grace and source timeouts never extend the observed duration. This remains an estimate from observation cadence, not proof of continuous physical presence between frames.

Spatial relation state is isolated by source, camera and group. Replayed terminal groups cannot open another occurrence. Group producers attach a runtime epoch and monotonic lifecycle sequence, allowing the shared terminal-subject ledger to compact closed groups without dropping replay protection after 4,096 occurrences. A new producer epoch can reuse a closed identifier; this does not establish person identity across sessions or qualify recovery when the previous group never closed.

Proximity observations require valid, recent capture evidence. Dwell follows media time rather than processing delay; same-frame member updates may update geometry but cannot add elapsed time. Older capture generations/sequences are rejected. A capture reset closes the old relation and starts fresh confirmation; a late close from the old generation cannot terminate the new relation. Publication freshness does not prove physical camera latency or synchronized remote clocks.

Dependencies

• The default toposync application bundle includes the first-party ONNX Runtime CPU stack.
• The GPU-oriented first-party bundles are published separately:
  • toposync-vision-cuda
  • toposync-vision-directml
• The official extension package now also includes the first-party tracker stack:
  • byte_world
  • simple_iou_kalman
  • norfair

Notes

• vision.detect resolves ModelManifest entries and builds an ONNX Runtime backend automatically when runtime=onnxruntime.
• By default, the ONNX Runtime backend prefers CPUExecutionProvider. Optional acceleration is opt-in via the toposync-vision-cuda / toposync-vision-directml bundles or TOPOSYNC_VISION_ONNXRUNTIME_PROVIDERS.
• Manifest files can be loaded from TOPOSYNC_VISION_MANIFESTS_DIR or TOPOSYNC_VISION_MANIFEST_PATHS.
• Custom manifests imported from the UI are persisted under .toposync-data/vision-manifests/ for the selected processing server.
• Built-in first-party manifests ship inside the wheel, but their ONNX weights do not. When no checkout-local artifact exists, official model ids resolve to TOPOSYNC_DATA_DIR/vision-models/... (or .toposync-data/vision-models/... by default).
• The extension now ships a built-in RTMDet detection shortlist in extensions/vision/manifests/:
  • rtmdet_det_tiny
  • rtmdet_det_small
  • rtmdet_det_medium
• The extension now also ships a built-in RF-DETR detection shortlist in extensions/vision/manifests/:
  • rfdetr_det_nano
  • rfdetr_det_small
  • rfdetr_det_medium
• In a source checkout, existing local artifacts under extensions/vision/models/... are still honored.
• In installed environments, official artifacts belong under the managed model store in TOPOSYNC_DATA_DIR/vision-models/... and are intentionally not bundled in the published package.
• The validated manual provisioning flow is documented in docs/VISION_MODEL_PROVISIONING.md.
• The initial assisted-provisioning foundation for RTMDet detection is already exposed in catalog metadata:
  • upstream checkpoint/config/metafile/paper links
  • planned local builder backend
  • planned supported platforms
  • explicit-consent requirement
• The UI can also trigger installation for models that have an admin-configured source. Supported source env vars are:
  • TOPOSYNC_VISION_MODEL_SOURCE_<MODEL_ID>
  • TOPOSYNC_VISION_MODEL_URL_<MODEL_ID>
  • TOPOSYNC_VISION_MODEL_PATH_<MODEL_ID>
  • TOPOSYNC_VISION_OFFICIAL_MODEL_SOURCE_DIR
  • TOPOSYNC_VISION_OFFICIAL_MODEL_BASE_URL
• RTMDet detection now also has an experimental assisted local build path:
  • Linux only in this phase
  • requires a local container runtime (docker or podman) on the processing server
  • still downloads the upstream checkpoint directly on that machine
  • still validates the exported end2end.onnx against the manifest checksum
  • now records provenance per job, including actor, accepted upstream sources, builder metadata, and final ONNX sha256
  • still keeps manual upload as the stable fallback path
  • can be started from the model recovery card or the Processing Servers screen
• RF-DETR detection now also has an experimental assisted local build path:
  • supports linux, darwin, and windows hosts in this phase
  • uses a host Python builder (rfdetr[onnx]) instead of MMDeploy
  • downloads the upstream checkpoint directly on that machine before exporting the ONNX locally
  • keeps manual ONNX upload available as the stable fallback path
  • is prioritized in the operator UI when nothing is installed and the local builder is actually available on that machine
• Remote download sources are only enabled when the manifest explicitly allows redistribution. The current built-in RTMDet/RTMDet-Ins manifests do not, so the safe first-party flow is local admin-managed copy.
• Product policy: RTMDet and RTMDet-Ins stay on guided_upload for now. RF-DETR is available only through assisted local build in this phase; it is not mirrored, bundled, or redistributed by Toposync.
• Product policy: Ultralytics/YOLO is not part of the official first-party vision runtime path.
• The extension now also ships a built-in RTMDet-Ins segmentation shortlist:
  • rtmdet_ins_tiny
  • rtmdet_ins_small
  • rtmdet_ins_medium
• RTMDet manifests use the dedicated mmdet_rtmdet parser, letterbox preprocessing, and COCO-80 labels.
• RF-DETR manifests use the dedicated rfdetr_detr parser and the official DETR-style dets + labels ONNX outputs.
• RTMDet-Ins manifests use the dedicated mmdet_rtmdet_ins parser and produce real binary mask artifacts.
• The processing server status now exposes:
  • heuristic hardware recommendations
  • runtime upgrade suggestions for CUDA / DirectML bundles
  • a per-task model catalog with availability (available, manifest_only, incompatible)
  • installation capability and progress for models that can be fetched/copied automatically
  • badges such as recommended, fastest, best_quality, edge
• vision.detect can emit finite per-detection events (emit_mode="events"), filter the stream to packets that contain detections (emit_mode="filter"), or keep every frame annotated (emit_mode="annotate").
• vision.track is now first-party and detector-agnostic: it consumes payload["vision"]["detections"].
• Every TrackedObject now carries camera_id, and can optionally carry world_anchor plus appearance_embedding_artifact_name for future multi-camera association work.
• vision.segment_instances writes payload["vision"]["segmentations"], attaches mask artifacts when enabled, and exposes the top mask as the semantic image key mask.
• vision.pose_estimate consumes existing person detections, runs the selected pose model, and publishes named, unclipped landmarks. The mediapipe_pose_33 manifest provisions a pinned ONNX artifact on the selected processing server.
• rtmpose_halpe26 is an experimental 2D alternative with an explicit Halpe26 skeleton. Its guided flow requires extracting end2end.onnx from the pinned upstream ZIP and uploading the ONNX, not the archive. The upload validates the pinned checksum and backend contract before replacing an existing artifact. Toposync does not bundle, mirror or automatically download these weights.
• RTMPose preserves raw SimCC scores, including values above one; they are not calibrated probabilities. The legacy pose-level score is the source detection score, explicitly identified in metadata, because this model does not output global pose confidence. Halpe26 does not supply relative 3D or a world pointing direction.
• Pose annotations retain their originating detection through vision.track; subject-specific processing does not rerun all people. Legacy keypoints remain available.
• vision.gesture_recognize adds time-based hand-raised, both-hands-raised, waving and pointing-candidate states. A two-dimensional pointing candidate is not a three-dimensional target or authorization to act.
• Setup, payload semantics, camera geometry requirements and experimental limitations are documented in Pose, feet and gestures. These capabilities are not qualified for unattended physical actions.
• ModelManifest now also accepts optional capabilities such as reid, so future re-identification models can be cataloged without changing the registry shape.
• The pipeline editor now chooses models by task, not framework/vendor id. Basic setup is guided for common users, while advanced details expose runtime/model internals and custom manifest import when needed.
• The first-party tracking backends are:
  • byte_world (primary): ByteTrack-style high/low confidence association with optional world-anchor matching.
  • simple_iou_kalman
  • norfair
• simple_iou_kalman and norfair remain useful benchmark baselines, but camera presets and recipes should use byte_world.
• vision.track emits stable per-object lifecycle packets directly. The product identity is payload["subject"]["id"]; technical tracker details stay available as tracklet_id, tracklet_ids, raw_tracking_id, and identity_id.
• Frame annotations use payload["vision"]["detections"], payload["vision"]["tracks"], and payload["vision"]["segmentations"]. Consumers that need one visual subject from an annotated frame should prefer tracks, then detections, then segmentations.
• vision.group_events consumes individual subject.type="event" packets and emits aggregated subject.type="group_event" packets with group_event_id, member_event_ids, category_summary, and grouped subject.bbox01 for quieter storage and notifications.
• A member CLOSE publishes a group UPDATE with that member inactive immediately, bypassing the periodic update interval. It does not close the group session; consumers can end relations without waiting for group idle timeout. Repeated member CLOSE packets do not publish another update.
• Spatial relations expose visits for the currently followed pair: confirmed approaches after observed separation, within visit_window_seconds (default 300). Initial proximity is not an observed entry. Membership loss, capture reset, participant change and timeout clear history; short departures within closing grace remain one occurrence. Each group retains at most 256 entries and reports a lower bound on overflow. This does not identify people across tracking sessions or infer intent. New person/vehicle presets include repeated approaches in their notification description; existing custom descriptions remain unchanged.
• Spatial relations require fresh capture evidence from each matched member in the same capture generation. Group publication does not renew member evidence. Dwell advances using the older member media timestamp and pauses when either member is unavailable; this estimates observed continuity and does not certify what happened between frames.
• Group identifiers and streams include the producer lifecycle epoch. A restarted producer creates independent groups even when its display counter starts at 1; it does not inherit proximity duration or assert continuity with the previous producer. Older groups close independently through their lifecycle or the consumer stale timeout. Consumers must treat identifiers as opaque.
• vision.detect events are short OPEN/CLOSE notifications; use vision.track when you need temporal identity, movement, and long-lived per-object lifecycle.
• vision.crop_objects supports preserve_scene: true to retain the selected image while attaching the projected subject context; it avoids a second image copy and keeps references outside the subject box visible. Default behavior still crops the subject.
• Object crops derived from subject.bbox01 carry subject_context with subject identity, original and crop-relative bounding boxes, and capture evidence. Crop-relative coordinates use the actual pixel boundaries after rounding, reusing the crop/warp projection path. Detection fallback does not claim subject binding.
• vision.crop_objects crops the bbox from payload["subject"]["bbox01"] by default and preserves event/subject identifiers. It is not instance segmentation.

Experimental visual history backend

Automatic installation requests also verify the checksum of an existing artifact before reporting it ready. This runs in the cancellable installation job, without copying or downloading the file again. A mismatch preserves the existing bytes and requires an explicit reinstall; pose models additionally retain their runtime validation.

The AI history endpoint accepts mode=visual for multilingual text-to-image ranking of visible notification images. It reuses notification authorization, time filters and SQLite. Similarity does not confirm identity or trigger automation. The notification filter offers words, text meaning and images. The four-file model set is available to the existing model installer as clip_multilingual_history; the simplified setup action in the search interface is still pending.

The Vision extension declares the tokenizer and safetensors reader dependencies. Its model manifest groups four pinned artifacts under the installation's vision-models directory. Explicit installation uses one cancellable job, verifies every file and reuses intact local artifacts. A failed transfer keeps earlier verified files for a subsequent attempt. Catalog availability requires all four checksums; search never downloads models. The image encoder is Xenova CLIP ViT-B/32 and the aligned text encoder is Sentence Transformers multilingual CLIP. The current quantized text export was validated on ARM64 CPU; other platforms need their own validation.

Search indexes up to eight images per request from the latest 250 matching notifications, with a 64 MiB image scan budget and two inference threads. Sessions are released after each request. The response reports incomplete indexing, unavailable images, scope truncation and fallback to literal text when the optional backend is unavailable. Changed or deleted evidence is rechecked before returning cached results. Model checksums and preprocessing version isolate the vector space. This bounded backend does not certify retrieval accuracy, all historical records, or unattended safety use.

Future runtime compatibility

• Pipeline configs must stay task-oriented and model-oriented. Do not add runtime, device, delegate, or vendor-specific toggles such as use_coral to vision.detect, vision.classify_image, or related operators.
• New inference stacks should be introduced as optional runtime backends plus ModelManifest entries. The manifest owns runtime, artifact_format, input.dtype, hardware accelerators, acquisition metadata, and provenance.
• Do not reuse an existing model_id for a materially different artifact/runtime. For example, an Edge TPU/TFLite artifact should get its own model id rather than replacing an ONNX model id.
• Runtime-specific upload, build, and install flows should remain disabled with clear diagnostics until the backend and artifact validation are implemented.