Skip to main content
sima-vision: live YOLO computer vision on a SiMa Modalix DevKit 3.0

SiMa.ai Palette SDK Neat

CI PyPI Python License YOLO26

Computer vision applications on the SiMa.ai Modalix DevKit 3.0. Object detection, instance segmentation, fall detection and retail analytics: one pipeline, no setup step. Needs a Modalix DevKit 3.0 and Python 3.10 or later.

Animated object detection icon Animated segmentation icon Animated fall detection icon Animated shopping trolley icon
Object detection Instance segmentation Fall detection Retail analytics
detect segment fall ret-analytics

Quickstart

on the DevKit

Every command here is typed on the board.

pip install sima-cli

# Login via SiMa.ai Portal
sima-cli login

# Install Neat core, once
sima-cli neat install core@v0.3.0

# Install sima-vision
pip install sima-vision

# Run object detection
sima-vision detect

The first run fetches a pretrained YOLO26 pack and a demo clip into ./assets, then runs. Every run after that reuses them. Both come from one public GitHub release, so getting them needs no login and no sima-cli.

App Fetched for you Writes
detect yolo26n-det-bf16-mla_tess-b1.tar.gz (21 MB) + a 1080p demo clip (13 MB) detections.mp4
segment yolo26n-seg-bf16-mla_tess.tar.gz (24 MB) + the same clip segmentation.mp4
fall the detection pack again + a shorter clip (1.2 MB) falls.mp4
ret-analytics the detection pack + shopping-trolley-items.mp4 retail.mp4

Each app writes annotated video. Stills are off unless you ask for them with --save.

 Instance segmentation  ·  per-pixel masks, with the background blurred
sima-vision segment                        # background blurred, instances sharp
sima-vision segment --keep-classes person  # only people stay sharp
sima-vision segment --anonymise            # the other way round: blur the people
sima-vision segment --no-blur              # a plain overlay, nothing blurred
 Fall detection  ·  tracks people and relabels the box when one goes down
sima-vision fall
sima-vision fall --classes person --confirm 2.0

The frame is the one detect draws: same palette, same boxes, same captions. A confirmed fall changes one thing, the class the box is labelled with, so a person reading person 0.93 a second earlier now reads FALL 0.93. Each fall is also printed on the console and counted in the run summary.

A confirmed fall re-arms only after sustained recovery.

 Retail analytics  ·  displays items in a shopping trolley
sima-vision ret-analytics
sima-vision ret-analytics --source shopping-trolley-items.mp4
sima-vision ret-analytics --classes bottle banana apple

Displays detected trolley items with boxes, names and confidence, and writes retail.mp4. Uses the same source, model and recording flags as detect. The white side menu uses a table with Qty, Item, Unit $, and Total $ columns, larger text, alternating rows, and an estimated basket total. Each product row is bold and has its own consistent color. Products enter the menu after three consecutive detections and stay in first-added order. Quantities increase only after a higher simultaneous count is confirmed for three frames. Brief misses never remove rows or reduce counts, and repeated frames never add purchases. The menu retains confirmed items for the run; removals are not inferred, and new items hidden behind existing ones cannot be counted reliably. All model classes are displayed by default; --classes optionally filters them. The default sample is shopping-trolley-items.mp4, fetched automatically. The default model recognizes COCO categories such as bottle, banana, apple, and orange. Product-specific recognition needs a trained model and matching labels. Point the camera at the trolley so the displayed detections represent its contents.

The optional retail.classes config setting selects item names or ids; an empty list displays all model classes.

Prices are rough per-item US estimates, not live quotes. Produce estimates use US retail listings as a reference. Unpriced classes show price N/A and are excluded from the known subtotal. Defaults include cherry ($0.10 each), cantaloupe ($3.00 each), banana ($0.20 each), cucumber ($0.75 each), bread ($2.00 per loaf), snack ($1.50 per small packet), orange ($1.00 each), avocados ($1.00 each), egg ($0.25 each), and milk ($4.00 per gallon container). Both avocado and avocados labels are supported. Override estimates for your products in config.yaml:

retail:
  classes: []
  prices:
    banana: 0.20
    tomato: 0.25
    apple: 1.00

The COCO model does not have a tomato class; tomato detection requires a model trained for tomatoes and a matching labels file.

If an automatically discovered config.yaml contains fall settings without a retail section, retail uses its own sample and output paths and prints a warning. CLI flags still win; an explicit --config uses the named file as written.

Apps arguments

Every flag, and which apps take it. All of them are for the four apps, which run on the DevKit. sima-vision <app> --help prints the same list.

Flag Apps What it does
--source, -s URI all File, https URL, RTSP URL, or empty for the sample clip
--source-type all video, rtsp or usb. Default video
--fps N all Source frame rate. Default 0, read from the stream
--width PX all Source width. Default 0, read from the stream's SPS
--height PX all Source height. Default 0, read from the stream's SPS
--model, -m PATH all Compiled pack, or an https URL to one. Empty fetches this app's default
--labels PATH all Newline-separated class names. Defaults to the packaged COCO list
--family NAME all Detection head. Must match the model or every box is noise
--conf T all Minimum confidence. Default 0.30
--iou T all Non-max suppression IoU. Default 0.60
--max-det N all Top-K cap per frame. Default 50
--frames, -n N all Stop after N frames. Default 0, runs until interrupted
--timeout MS all How long to wait for a frame before giving up. Default 20000
--video-path PATH all Where the annotated recording is written
--no-video all Do not record
--video-encoder all sima (hardware H.264, default) or opencv (software, about ten times slower)
--video-bitrate KBPS all Target bitrate for the hardware encoder. Default 12000
--save all Also write annotated stills. Off by default, so only the video is written
--save-dir DIR all Where annotated stills are written. Implies --save
--save-every N all Write every Nth still. Default 10 once stills are on; implies --save. 0 disables
--no-save all Do not write stills. Already the default; use it to override a config file
--show-fps all Show the FPS badge on the overlay (hidden by default)
--no-hud all Leave the frame-rate badge off the overlay
--hud-scale N all Frame-rate badge font size. Default 2.4, which is 1.5x the caption scale; 0 follows the caption scale exactly
--hud-thickness N all Badge stroke weight. Default 6, which is 1.5x the caption thickness; 0 follows it exactly
--hud-bg B,G,R all Badge fill colour. Default 132,28,193 (#C11C84)
--hud-color B,G,R all Badge text colour. Default 255,255,255
--hud-padding PX all Gap between badge text and its edge, which is what sizes the badge. Default 22; 0 follows the caption padding
--config, -c PATH all Config file. Defaults to ./config.yaml
--no-config all Ignore any config file and use defaults plus these flags
--validate all Resolve and check the settings, then stop. Needs no board
--quiet, -q all Warnings, errors and the closing report only
--profile all Per-stage timings, when a run is slower than it should be
--queue-depth N all Neat's own queue depth. Below 4 the graph drops frames when the recorder falls behind. Default 4
--sink-queue-depth N all Finished frames that may wait for the recorder. Host memory only. Default 12
--sink-queue-mb MB all Memory budget for that backlog, which grows it to fit a known clip
--output-buffers N all Buffers each public output may hold. Default 1
--decoder-buffers N all Buffers to ask the decoder for. Default 0, sized from the stream's reference frames
--decoder-tuning all Hardware decoder preset. Default default, which keeps every frame; auto drops frames in bursts
--segment-frames N all Frames per piece when a clip is too long for one decode. Default 150; 0 runs it whole
--blur / --no-blur segment Blur the background and keep instances sharp. On by default; --no-blur draws a plain overlay
--blur-method segment gaussian, pixelate or none. Default gaussian
--blur-strength PX segment Gaussian kernel width at 1080p. Default 41
--keep-classes CLASS... segment Names or ids that stay sharp. Default: every detected class
--anonymise, --anonymize segment Blur the instances instead of the background
--mask-threshold T segment Mask cut-off as a probability. Default 0.5; lower grows instances
--no-masks segment Blur around plain boxes instead. Needs no segment head
--minimal segment Pull frames and do nothing else. Tells a slow app from a stalled graph
--classes CLASS... fall, ret-analytics Fall classes (default person) or retail products (default all model classes)
--confirm S fall How long a fall signal must hold before it counts. Default 1.5
--no-fall fall Track without judging falls, which is how you tune tracking first

A config.yaml in the working directory is picked up on its own. Flags beat it, and it beats the built-in defaults.

Your own model

on your PC

A trained .pt has to be compiled into a .tar.gz pack before the board can run it. The compiler is the afe package and exists only inside the Palette Model SDK container, which is x86 Docker, which on Windows means WSL2. So the whole of this runs on your PC, needs a community.sima.ai account and ~10 GB of disk, and takes about 30 minutes the first time. Quickstart needs none of it.

 Converting a trained .pt into a DevKit pack
# --- PowerShell, as Administrator -------------------------------------------
wsl --install -d Ubuntu
wsl -l -v                             # want: Ubuntu, Running, 2

# Install Docker Desktop, then Settings > Resources > WSL integration > Ubuntu.
# Without that toggle docker works in PowerShell and is missing inside Ubuntu.

wsl -d Ubuntu                         # everything below is Ubuntu. exit comes back

# --- Ubuntu ------------------------------------------------------------------
docker run hello-world                # must print "Hello from Docker!"

sudo apt update && sudo apt install -y python3-venv python3-pip
python3 -m venv ~/sima                # Ubuntu refuses a bare pip install
source ~/sima/bin/activate            # new shells start outside it. Look for (sima)
pip install sima-cli
sima-cli login
sima-cli install ghcr:sima-neat/sdk:v2.0.0    # several GB, once per machine

cd /path/to/your/model                # the folder holding best.pt
sima-cli sdk setup --workspace .      # . is what gets mounted. 15-20 minutes
activate-model-compiler

# --- Model SDK shell ---------------------------------------------------------
pip install "sima-vision[compile]"    # ultralytics, onnx, onnxsim
sima-vision compile best.pt           # ONNX, bfloat16, tessellate, ELF. 10-15 minutes
                                      # -> build/best_mpk.tar.gz

# --- on the DevKit -----------------------------------------------------------
sima-vision push build/best_mpk.tar.gz
sima-vision detect --model best_mpk.tar.gz
sima-vision detect --model https://example.com/my-model.tar.gz   # a URL works too

Moving files

on your PC

Output lands beside the run, on the board, and both commands are typed on your PC. Name the board once and neither command needs --host:

$env:SIMA_VISION_DEVKIT="sima@<devkit-ip>"
sima-vision pull                     # everything the run left
sima-vision pull --into results/
sima-vision push my-clip.h264

The default pull includes retail.mp4 and retail-report.json when present, along with the other apps' outputs. Missing files are skipped.

Environment

Variable What it does
SIMA_VISION_DEVKIT The board, as user@address, so push and pull stop asking
SIMA_VISION_ASSETS Where clips and models are downloaded. Default ./assets
SIMA_VISION_PYNEAT The pyneat virtualenv, when the search does not find it
SIMA_VISION_PYNEAT_INDEX A pip index carrying a pyneat wheel, if your site publishes one
SIMA_VISION_AUTO_INSTALL 0 to look but never install
SIMA_VISION_QUIET Non-empty is --quiet for every command
SIMA_VISION_COLOR 0 or 1 to force colour off or on. NO_COLOR also works

Contributing

git clone https://github.com/RizwanMunawar/sima-projects.git
cd sima-projects
pip install -e ".[dev]"

ruff check sima_vision tests
pytest -q

The tests need no board.

License

The models used here for testing are Ultralytics YOLO26, under AGPL-3.0. All other parts of this repository are under Apache-2.0. See LICENSE.

Credits

  • SiMa.ai for Modalix, the Palette SDK and Neat
  • Ultralytics for the YOLO26 models
  • Timothy Do for the Neat 0.4.0 port, frame-complete recordings and hardware H.264 recording

Built by Muhammad Rizwan Munawar. If this saved you an afternoon, star the repo and pass it on to someone else bringing up a DevKit.

GitHub    LinkedIn    X    YouTube    Medium

Metadata

Release files for sima-vision 26.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sima-vision 26.0.4
File Size Uploaded Provenance
sima_vision-26.0.4.tar.gz 250.5 kB With provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for sima-vision 26.0.4
File Interpreter ABI Platform
sima_vision-26.0.4-py3-none-any.whl Python 3 none any Details

Total release size: 429.7 kB

Release files / sima_vision-26.0.4.tar.gz

Download URL sima_vision-26.0.4.tar.gz
Size 250.5 kB
Tags Source
SHA-256 checksum
How to use checksums
bfaefbe2bfbd489ec4d4b273fcd6a3c671eec88df3cf95292318c927e36bd288
BLAKE2b-256 checksum
How to use checksums
2a7beb0baf6195d74460d46c4fa2cacd761cbcbbbe972a0f2e25bc80b3fb9786
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.

Transparency log

Release files / sima_vision-26.0.4-py3-none-any.whl

Download URL sima_vision-26.0.4-py3-none-any.whl
Size 179.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
24af5060465ebaaaefe61545eefc2d128b3319f7ba850248541f36eeadb4e9c5
BLAKE2b-256 checksum
How to use checksums
ccfd357f9be4d8301b790f9ea1270ce92967074742b2b9cf2f58be1b53877809
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.

Transparency log

Security for sima-vision 26.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 2 files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sima_vision-26.0.4.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page