vigilo-stream

Zero-copy multi-modal stream fusion engine for real-time AI pipelines in Python.

vigilo-stream provides Python bindings for the stream fusion engine in vigilo-core. It gives Python vision and proctoring pipelines direct access to video frames and temporal rule evaluation without copying memory across the FFI boundary.

• Zero-copy buffer sharing: Frame memory allocated in Rust is exposed directly to NumPy and PyTorch through __array_interface__ and the buffer protocol.
• Lock-free frame exchange: Capture workers publish frames through ArcSwap slots, discarding stale frames automatically instead of building queues.
• Deterministic temporal fusion: The FusionEngine processes detection signals through configurable hysteresis bands, hold timers, and score accumulators. Given the same input, replay produces identical events.
• Multimodal detection: Wraps the vigilo-core inference pipeline for face detection (YuNet), head pose (MobileNetV3), gaze estimation (MobileGaze), object detection (YOLOX-Nano), and identity matching (ArcFace).

Installation

pip install vigilo-stream

You can import the library using either vigilo_stream or the rustream alias.

Quick start

import vigilo_stream
import numpy as np

# 1. Zero-copy frame operations (no neural model files required)
frame = vigilo_stream.create_synthetic_frame(1280, 720, seq=1, r=255, g=0, b=0)
print(frame.width, frame.height, frame.shape) # 1280 720 (720, 1280, 3)

# Expose Rust memory directly as a NumPy array without copying
arr = np.asarray(frame)
assert arr.__array_interface__["data"][0] == frame.__array_interface__["data"][0]

# 2. Vision and proctoring pipeline
# Pipeline automatically downloads default model weights on first run
with vigilo_stream.Pipeline(models_dir="models") as pipe:
    pipe.start("camera:0")  # Accepts "camera:0", "file:clip.mp4", or "dir:frames/"

    while pipe.is_running():
        frame = pipe.poll_frame()
        if frame:
            img = np.asarray(frame)

        snapshot = pipe.snapshot()
        if snapshot:
            print(f"Faces: {snapshot.face_count}, Pose: {snapshot.head_pose}")

        events = pipe.events()
        for event in events:
            print(f"Violation: {event}")

# 3. Headless deterministic stream fusion (no neural models or camera required)
engine = vigilo_stream.FusionEngine()
events = engine.replay("recorded_session.jsonl")
print(f"Replayed session produced {len(events)} events.")

Model weights

The neural pipeline uses ONNX Runtime models:

• Face detection: YuNet (face_detection_yunet_2023mar.onnx)
• Head pose: MobileNetV3 (headpose_mobilenetv3_small.onnx)
• Gaze estimation: MobileGaze (mobileone_s0_gaze.onnx)
• Object detection: YOLOX-Nano (yolox_nano.onnx)

By default, Pipeline(models_dir="models") downloads missing models on first use. You can also download them explicitly:

import vigilo_stream
vigilo_stream.download_models("models")

Alternatively, download them using curl:

mkdir -p models
curl -sSL -o models/face_detection_yunet_2023mar.onnx https://github.com/opencv/opencv_zoo/raw/main/models/face_detection_yunet/face_detection_yunet_2023mar.onnx
curl -sSL -o models/headpose_mobilenetv3_small.onnx https://github.com/yakhyo/head-pose-estimation/releases/download/weights/mobilenetv3_small.onnx
curl -sSL -o models/mobileone_s0_gaze.onnx https://github.com/yakhyo/gaze-estimation/releases/download/weights/mobileone_s0_gaze.onnx
curl -sSL -o models/yolox_nano.onnx https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_nano.onnx

Architecture

Camera / Video File / Image Directory
                 │
                 ▼
  FrameSource (DirectShow / FFmpeg)
                 │
                 ▼
     ArcSwap Latest-Frame Slot  ◄── Zero-copy pointer sharing with NumPy
        ┌────────┴────────┐
        ▼                 ▼
   Face Worker      Object Worker
  YuNet+Pose+Gaze     YOLOX-Nano
        └────────┬────────┘
                 ▼
              Signals ──► FusionEngine ──► Events / Violations

Building from source

Requirements:

• Rust 1.80 or newer
• Python 3.9 or newer
• C++ build tools (MSVC on Windows, GCC/Clang on Linux and macOS)

# Set up a virtual environment and install build tools
uv venv
uv pip install maturin pytest numpy

# Build and install the extension into the active environment
uv run maturin develop

# Run the test suite
uv run pytest -v tests/

Release notes

v0.1.1

• Added download_models() helper to fetch default ONNX model weights automatically.
• Enhanced Pipeline to download missing model files automatically on first use (auto_download=True).
• Added MODEL_URLS mapping and updated documentation.

v0.1.0

• Initial release of vigilo-stream (with rustream backward-compatibility alias) targeting Python 3.9 through 3.13.
• Implemented Frame with __array_interface__ and memoryview() support for zero-copy NumPy interop.
• Implemented FusionEngine with single-frame stepping and deterministic JSONL log replay.
• Implemented Pipeline context manager wrapping camera capture, detection workers, and event polling.
• Added data bindings for BBox, FaceDetection, HeadPose, Gaze, ObjectDetection, Signals, Violation, and Event.
• Multi-platform CI testing across Windows, Ubuntu, and macOS.

License

AGPL-3.0. See LICENSE for details.