VisionServeX

License-aware local computer-vision gateway — curated models, honest blockers, no fake claims.
Serve modern CV models on your machine. Local-only by default. No data retained.

What is VisionServeX?

VisionServeX is an open-source Python framework for running modern computer vision models locally and exposing them through a stable HTTP API. It works as a local model gateway: start it once, call any supported model through one clean API, on your own hardware, with no data leaving your machine.

Every model in the registry carries an explicit license classification, an honest availability status, and a clear commercial posture. Blockers are documented — not hidden.

Commercial-safe by default: restricted models (research-only, non-commercial, AGPL/copyleft, legal-review, BYO-license) are not enabled unless you explicitly choose a research/BYO pathway and acknowledge the applicable restrictions. A model is only commercial-safe when its code AND weights are verified permissive — code license alone is never enough. See docs/model_policy.md. MedSAM2, for example, is research-only and is never commercial-safe.

Why VisionServeX?

• Local-first inference — binds to 127.0.0.1 by default; images never leave your machine
• License-aware registry — every model is classified: commercial-safe core, BYOT gated, non-commercial, or external-API-only
• Stable Python + CLI + HTTP API — one interface across detection, segmentation, embedding, depth, classification, and open-vocabulary tasks
• Commercial-safe core — SAM v1/2/2.1, DINOv2, RF-DETR, Florence-2, CLIP, OWLv2, Grounding DINO, and more; all Apache-2.0 or MIT, no token required
• Permissive detector training (v3.13/v3.14) — fine-tune LibreYOLO (YOLOX / YOLOv9 / RT-DETR / D-FINE) on your own YOLO datasets, reload trained checkpoints, and export to ONNX — no Ultralytics/AGPL. See docs/libreyolo_training.md. Standalone HF D-FINE stays inference-only; YOLO-NAS (non-commercial) is excluded.
• Classic torchvision classifiers (new in v3.15.0) — AlexNet / ResNet / ResNeXt / Wide-ResNet / DenseNet / MobileNet / EfficientNet / ConvNeXt: pretrained ImageNet inference, ImageFolder fine-tune, checkpoint reload, and ONNX export. BSD-3-Clause, commercial-safe. See docs/torchvision_classifiers.md.
• Capability truth contract (new in v3.15.0) — VisionModel(id).capabilities() returns an honest per-model object (legal status, inference/train readiness, reload+predict, export); no fake-ready states. Full inventory in docs/qa/v315_model_coverage/.
• BYOT support for gated models — SAM3/SAM3.1, DINOv3; your token, your cache, your accepted license
• No bundled gated weights — VisionServeX never puts gated model weights into PyPI, GitHub, Docker, or any release artifact
• Honest blockers — unavailable models explain exactly why and what to do next

Quickstart

pip install 'visionservex[hf,rfdetr]'

visionservex --version
visionservex getting-started        # personalized guide

# Detection (RF-DETR, no token needed)
visionservex detect rfdetr-small image.jpg

# Segmentation (SAM2.1, no token needed)
visionservex sam-family smoke-test sam2.1-hiera-small image.jpg

# HTTP gateway
visionservex serve                  # http://127.0.0.1:8080
curl -F "image=@image.jpg" -F "model_id=rfdetr-small" http://127.0.0.1:8080/detect | jq

from visionservex import VisionModel, VSX

# Direct inference (no server)
result = VisionModel("rfdetr-small").predict("image.jpg")
result.to_json()

# SAM2 segmentation
VSX.sam("sam2.1-hiera-small").segment("image.jpg", box=[10, 20, 200, 220])

# DINOv2 embedding (Apache-2.0, no token)
VSX.dino("dinov2-base").embed("image.jpg")

What works today

Hugging Face BYOT for gated models

SAM3, SAM3.1, and DINOv3 are gated on Hugging Face. VisionServeX provides a clean BYOT path: you supply your own token, accept the upstream license once, and the weights stay in your local HF cache.

# Step 1 — connect your token
huggingface-cli login                        # or:
visionservex hf connect --token-env HF_TOKEN

# Step 2 — check status (no download)
visionservex hf status
visionservex hf check-model facebook/sam3

# Step 3 — accept the upstream license on the model page, then pull
visionservex model pull sam3-base --accept-upstream-license
visionservex model doctor sam3-base

from visionservex import VSX

VSX.hf.status()
VSX.model("sam3-base").pull(accept_upstream_license=True)
VSX.sam("sam3-base").segment("image.jpg", text="person")   # BYOT inference
VSX.dino("dinov3-vitb16").embed("image.jpg")               # BYOT embedding

SAM3/SAM3.1 real mask results (v3.10.0)

Real mask artifacts are produced after accepting the upstream license. Both models have been benchmarked locally and produce confirmed non-zero masks:

VisionServeX does not redistribute gated model weights. Weights remain in your HF cache. They are never included in PyPI, GitHub, Docker, or any VisionServeX release artifact.

License and commercial posture

Commercial-safe core includes: SAM v1/2/2.1, DINOv2, D-FINE, RF-DETR family, Grounding DINO (open variants), Florence-2, CLIP, OWLv2, SigLIP2, SwinV2, MaxViT, depth-anything-small, and more — code and weights verified permissive. No token needed; weights download from official upstream sources on demand. (ConvNeXtV2 is excluded: upstream CC-BY-NC vs HF apache-2.0 conflict → research-only. See docs/model_policy.md.)

BYOT models (SAM3, SAM3.1, DINOv3) are not automatically commercial-safe. Commercial use depends on the upstream license you accepted. VisionServeX provides the infrastructure; the license decision is yours.

Detailed counts and bucket breakdowns: docs/global_model_count.md

Model Families

Models marked expert_sidecar require an isolated environment (OpenMMLab, Detectron2). Use visionservex openmmlab create-env for the exact conda recipe.

Models marked BYOT (gated) require accepting the upstream license on Hugging Face.

Python API

from visionservex import VisionModel, VSX, Client

# Direct inference (no server needed)
result = VisionModel("dfine-s-o365-coco").predict("image.jpg")
result.to_json()
result.save("outputs/")
result.plot()                          # PIL Image

# Context manager with GPU cleanup
with VisionModel("rfdetr-large", device="cuda") as model:
    result = model.predict("image.jpg")

# HTTP client
client = Client("http://127.0.0.1:8080")
result = client.detect("rfdetr-small", "image.jpg")
result = client.grounded_segment("grounded-sam2", "image.jpg", prompt="car, person")

Output normalization handles all common detection serialization formats:

from visionservex import normalize_detections

dets = normalize_detections([
    {"xyxy": [10, 20, 100, 200], "score": 0.9, "label": "cat"},
    {"box": {"x1": 10, "y1": 20, "x2": 100, "y2": 200}, "confidence": 0.8, "category": "dog"},
])

HTTP Gateway

pip install 'visionservex[server,hf,rfdetr]'
visionservex serve                     # http://127.0.0.1:8080

# Detect
curl -F "image=@image.jpg" -F "model_id=rfdetr-small" http://127.0.0.1:8080/detect | jq

# Segment
curl -F "image=@image.jpg" -F "model_id=sam2.1-hiera-small" \
     -F 'box=[10,20,200,220]' http://127.0.0.1:8080/segment | jq

Public mode (Cloudflare Tunnel):

export VISIONSERVEX_AUTH__ENABLED=true
export VISIONSERVEX_AUTH__API_KEY=$(visionservex gateway token 2>&1 | grep "API key:" | awk '{print $NF}')
visionservex tunnel config --domain api.yourdomain.com --out tunnel.yaml
visionservex serve &
visionservex tunnel run tunnel.yaml --i-understand-this-is-public

CLI Reference (selected)

# Model lifecycle
visionservex detect dfine-s-o365-coco image.jpg --conf 0.25
visionservex segment rfdetr-seg-medium image.jpg
visionservex classify swinv2-base image.jpg --top-k 5
visionservex open-vocab grounding-dino-swin-b image.jpg --prompt "car,person"
visionservex embed dinov2-base image.jpg --out embedding.npy
visionservex similarity siglip2-base-patch16-224 a.jpg b.jpg

# Model info
visionservex model pull rfdetr-small
visionservex model info rfdetr-small
visionservex model license sam3-base
visionservex model doctor sam3-base
visionservex model-card show dfine-s-o365-coco

# Capabilities and health
visionservex capabilities report
visionservex models health --runnable-only
visionservex recommend --task detect --goal accuracy

# HF BYOT
visionservex hf status
visionservex hf whoami
visionservex hf check-model facebook/sam3

# Benchmark (requires annotated dataset)
visionservex benchmark benchmark-competitiveness \
  --models dfine-s-o365-coco,rfdetr-small \
  --max-images 20 --device auto

# Debug
visionservex debug-output rfdetr-small image.jpg --threshold 0.01
visionservex dev resources
visionservex dev test quick

Feature Intelligence

# Build a similarity index
visionservex embed dinov2-base folder/ --out embeddings/
visionservex index dinov2-base folder/ --out indexes/dinov2_base
visionservex search dinov2-base query.jpg --index indexes/dinov2_base --top-k 10
visionservex deduplicate dinov2-base folder/ --threshold 0.98 --out dups.csv

# Surveillance search (index + text query)
visionservex video-search index ./frames/ \
  --detector owlv2-base-patch16 --embedder siglip2-base-patch16-224 \
  --prompt "person" --out indexes/camera01
visionservex video-search query indexes/camera01 --text "red shirt" --top-k 20

Privacy and Security

• Binds to 127.0.0.1 by default — nothing leaves your machine
• Images decoded in memory; never written to disk by default
• Log redaction removes tokens and API keys from all output
• No data retained between requests by default

VisionServeX cannot provide E2E encryption — the inference server must see plaintext image tensors. It provides local-first processing, no-retention defaults, and auth for public mode. See docs/privacy.md.

visionservex security audit --json
visionservex privacy inspect-cache
visionservex privacy cleanup --dry-run

Resource Safety

VisionServeX includes a resource guard (RAM / VRAM / disk) that prevents exhaustion during testing and development. Production CLI commands are unaffected.

visionservex dev resources
visionservex dev gpu-profile --format json
visionservex gpu guard-status
visionservex gpu cleanup --dry-run
visionservex dev kill-tests          # kill stray pytest processes

Default budgets: 8 GB free RAM, 2 GB free VRAM, 10 GB free disk. See docs/agent_safety.md and AGENT_RULES.md.

Installation

pip install visionservex                        # base (no heavy deps)
pip install 'visionservex[server]'              # + HTTP API server
pip install 'visionservex[hf]'                  # + HF Transformers (D-FINE, SAM, GD, SwinV2, …)
pip install 'visionservex[rfdetr]'              # + RF-DETR and RF-DETR-Seg
pip install 'visionservex[dino]'                # + DINOv3 depth head (transformers>=5.10)
pip install 'visionservex[server,hf,rfdetr]'    # full recommended

OpenMMLab (RTMPose, RTMDet-R, Co-DINO): Docker sidecar or pip install openmim && mim install mmengine mmcv mmpose. See docs/openmmlab_expert_models.md.

Known Limitations

• SAM3/SAM3.1: Gated on Hugging Face; requires accepting the upstream license. BYOT only.
• DINOv3 CHMv2 depth head: Requires transformers>=5.10; may conflict with Florence-2 (<5.0) — install in a separate env.
• Florence-2: Requires isolated env (transformers==4.46.3 + einops + timm). Use visionservex florence2 create-env for the validated recipe.
• DEIMv2: Registered but not wired — no HF Transformers support yet; custom loader required.
• MedSAM: Research only; non-commercial restricted. 2D promptable (box/point), inference only. See docs/medical_segmentation.md.
• MedSAM2: Research-only, non-commercial weights — not commercial-safe, not in the runtime registry, not reachable via predict/HTTP. An experimental real 2D runtime is available in an isolated env via visionservex medical medsam2 … (CPU-verified; 3D/video NOT wired). See docs/medical_segmentation.md.
• SAM2.1 ONNX: Image-encoder export works via Module shim. Full interactive decoder ONNX not yet verified.
• OpenMMLab (RTMPose, RTMDet-R/R2, Co-DINO, InternImage): Expert sidecar; use visionservex openmmlab create-env.
• Apple MPS: Implemented but not maintainer-verified.
• GPU: CUDA verified on RTX 5080 for 6+ model families. Run visionservex gpu smoke-test.

Documentation

Reports and Audits

Detailed audit evidence — policy matrices, benchmark ledgers, test run reports, execution logs — lives in docs/reports.md and notebook/99_final_report/reports/. It is internal audit data, not public marketing. The counts there (policy rows, manifest entries, test passes) are measurement evidence, not product-health scores.

See docs/reports.md and docs/global_model_count.md.

License and Model Licenses

Apache-2.0. See LICENSE and NOTICE.

Each integrated model retains its own upstream license. Review model, checkpoint, and dataset licenses before commercial use. See docs/model_licenses.md.

Citation

@software{sajjadi2026visionservex,
  author = {Arash Sajjadi},
  title  = {{VisionServeX: A license-aware framework for local CV model serving}},
  year   = {2026},
  url    = {https://github.com/arashsajjadi/VisionServeX},
  note   = {Developed under the supervision of Prof. Mark Eramian, University of Saskatchewan.}
}

Author: Arash Sajjadi — PhD Candidate, Department of Computer Science, University of Saskatchewan
Supervision: Prof. Mark Eramian, Computer Vision Lab
(This project is not an official product of the University of Saskatchewan.)