VisionServe Python Client

Prefer JavaScript/TypeScript? There's a sibling client with the same API in clients/js/.

A lightweight Python client SDK for the VisionServe HTTP server. It talks to the Go runtime over HTTP (default http://localhost:11435) — it is not the inference runtime and pulls no inference engine into Python. Think of it like Ollama's Python client.

The transport uses only the Python standard library (urllib), so the client has no required third-party dependencies. numpy and pillow are optional and only needed for:

• passing numpy.ndarray / PIL.Image images to predict(), and
• decoding masks with Mask.to_ndarray().

Contents

• Install
• Run the server
• Quickstart
• CLI
  • Output
  • Examples
• Public API
  • Image inputs
  • Client-side resizing (on by default)
  • Result types
  • Detection
  • Segmentation
  • Background segmentation
  • Open-vocab / Grounded-SAM
  • Depth estimation
  • Classification
  • CLIP embeddings
  • SCRFD / OCR
  • Grasp detection
  • Cameras and video — Client.watch()
• Post-processing
  • Grasp post-processing
  • Size filtering
  • Visualization
• Script examples
• Tests
• Changelog

Install

pip install visionserve               # from PyPI (recommended)
# or from source:
pip install -e clients/python
# optional extras for ndarray/PIL image inputs and mask decoding:
pip install -e 'clients/python[images]'

Run the server first

The client needs a running VisionServe server (which in turn needs the ONNX Runtime shared library at runtime). From the repo root:

make serve            # starts the Go server on :11435
make serve IDLE=0     # ...and never auto-unload models (keep them resident)

Slow first request after a pause? By default each model auto-unloads after 5 min idle (idle_unload_seconds: 300), so the next request pays a full reload (ONNX session re-create + CUDA init). Keep models resident with the serve flag --idle-unload-seconds 0 — via make serve IDLE=0, the binary (visionserve serve --idle-unload-seconds 0), or Docker (append it to the container command). -1 = use each manifest's value (default); N = override all to N s.

Quickstart

from visionserve import Client

c = Client()                       # http://localhost:11435, timeout=120s
print(c.health())                  # {"status": "ok"}

for m in c.list_models():
    print(m.name, m.task, m.license, m.state)

c.load("rf-detr")
res = c.predict("rf-detr", "cat.jpg")
print(res.task, res.duration_ms)
for d in res.detections:
    print(d.cls, d.conf, d.bbox)   # bbox = [x, y, w, h] in ORIGINAL image pixels

print([m.name for m in c.ps()])    # currently loaded models

CLI

Installing the package registers a visionserve console command. Like the SDK it is a thin HTTP client — it runs no inference itself, it just talks to a running VisionServe server (the Go binary visionserve serve, default http://localhost:11435). The bare command needs only the standard library; --save (annotated images) additionally needs Pillow:

pip install visionserve            # CLI works with stdlib only
pip install 'visionserve[images]'  # add Pillow for --save

Commands:

Global flags (accepted before or after the subcommand):

predict flags:

list / ps accept --json to print JSON instead of a table.

Output

predict prints the unified result as JSON to stdout (pipe-friendly; field names match the server wire schema, e.g. class, with empty arrays omitted). A one-line summary goes to stderr:

predict: model=rf-detr task=detection device=gpu:0  client=42.1ms server=12.3ms  (12 detections)

Here client is the wall-clock time around the predict() HTTP round-trip and server is the server's duration_ms (inference only). Both are captured before the image is drawn or saved, so visualization cost never inflates the reported latency. The --save filename <stem>.python.<model>.<task>.png embeds the client type (python), so outputs from the Python/JS/Go CLIs for the same image never collide.

Examples

# Start the server first (Go binary):
visionserve serve &

# Detection -> JSON on stdout, summary on stderr
visionserve predict rf-detr cat.jpg

# Open-vocab + save annotated image (auto-named cat.python.grounding-dino.open_vocab.png)
visionserve predict grounding-dino cat.jpg --prompt "cat. remote." --save

# Box-prompted segmentation, explicit output path
visionserve predict mobile-sam dog.jpg --box 50,40,200,180 --save-as dog_masks.png

# Open-vocab grasps, remote server, top-1 grasp per object
visionserve --host http://10.0.0.5:11435 predict grasp-gd bin.jpg --prompt "mug." --max-grasps-per-object 1 --save

# Background support-surface in a region, classical-CV method
visionserve predict background scene.jpg --method cv --roi 300,380,700,330 --save

# Detect only inside a region (any model); results come back in ORIGINAL coords
visionserve predict rf-detr scene.jpg --roi 100,100,400,300

# A camera or video, one line per frame (JSON lines with --json)
visionserve watch 0 --model rf-detr --fps 5 --track
visionserve watch clip.mp4 --model rf-detr --every-frame --json > detections.jsonl

# Registry / memory management
visionserve list
visionserve ps
visionserve load rf-detr

Public API

Client(host="http://localhost:11435", timeout=120, *, base64_arrays=False, resize="auto", jpeg=True, jpeg_quality=90)

Every predict option, what it does and which models read it, with real outputs: Clients › Python on the docs site.

min_size / max_size filter by bounding-box area as a percentage of the image area (0–100; 0 = no limit). Example: min_size=0.5 keeps only objects covering at least 0.5% of the image. The conversion to absolute pixels is done server-side using the uploaded image dimensions.

roi=[x, y, w, h] (ORIGINAL image pixels) crops to that region, runs the model on the crop only, and maps every result back to original coordinates — generic to all models (detection / segmentation / grasp / background). Box / point prompts are given in original coords and shifted into the crop automatically.

box_threshold / text_threshold are GroundingDINO tuning knobs (grounding-dino, grounded-sam, grasp-gd); None = use the server manifest/default. box_threshold is the score cutoff that filters detections; text_threshold is a second cutoff on the same score (it does not change labels, which are always a whole prompt phrase). See Open-vocab / Grounded-SAM.

method selects the background algorithm per request (auto, depth, sam, cv, automask; None = the server default, auto). bg_max_area / fg_min_area / grid_size are background tuning knobs (None = use the server manifest/default); bg_max_area and fg_min_area are percentages of the image area and only affect the sam / automask methods, while grid_size only affects automask. See Background segmentation. Do not use min_size / max_size with background — they filter the output bboxes and drop the surface mask (whose bbox spans the support surface).

Image inputs

predict(model, image, ...) accepts four image types — choose whichever fits your pipeline:

from visionserve import Client
from PIL import Image
import numpy as np

c = Client()

# 1. File path (str or os.PathLike) — simplest
res = c.predict("rf-detr", "photo.jpg")

# 2. PIL.Image — e.g. after augmentation or crop
pil_img = Image.open("photo.jpg")
res = c.predict("rf-detr", pil_img)

# 3. numpy ndarray — HWC uint8 (H×W×3), or float [0,1] scaled to uint8 automatically
arr = np.array(pil_img)           # shape (H, W, 3), dtype uint8
res = c.predict("rf-detr", arr)

# 4. raw bytes — already-encoded PNG/JPEG
with open("photo.jpg", "rb") as f:
    raw = f.read()
res = c.predict("rf-detr", raw)

Grayscale (H, W) ndarrays are automatically promoted to RGB. Float arrays in [0, 1] are scaled to uint8. Paths and bytes go out verbatim and PIL images / arrays as lossless PNG, except a photo larger than the model can use, which is shrunk and sent as JPEG by default (next section). Boxes, points and numeric options may also be numpy arrays / scalars.

Client-side resizing (on by default)

A model resizes every photo to its own small input on the server (RF-DETR 560 × 560, GroundingDINO 800 × 800, CLIP 224 × 224), so most of a 12-megapixel upload is thrown away. GET /api/models therefore tells the client, per model, how far a photo can be shrunk without changing what the model sees (max_useful_side bounds the longer side, max_useful_short_side the shorter, 2 × the model's input; None = never). By default the client fetches it once, shrinks a larger photo to it, sends JPEG (quality 90, 4:4:4 colour), scales box / point / roi into the sent photo and maps every box, mask and grasp back, so results are in your photo's pixels as before. Only a shrunk photo is re-encoded. res.client_resize records what was sent and why (reason; None = your bytes, untouched). With the server on this machine (localhost, 127.0.0.0/8, ::1) a photo is shrunk only when that at least halves its sides: on localhost the upload is free, and a milder shrink costs the client more to decode than the server saves (the loopback rule).

c = Client()                                   # resize="auto", jpeg=True, jpeg_quality=90
res = c.predict("rf-detr", "photo_4000x3000.jpg")
print(res.client_resize)    # ClientResize(original_width=4000, ..., sent_width=1493, sent_height=1120, jpeg_quality=90, reason='hint')

Client(resize="off")                           # send every photo as given (the 0.1.x behaviour)
c.predict("rf-detr", "photo.jpg", resize=1280, jpeg=False)   # per call: longer side 1280, PNG

Never resized: models without a hint (SAM family and every model returning masks, OCR, grasping, background, templates, crop-naming pipelines, depth-anything-v2), a photo already within the hint (sent exactly as given), and requests with depth, dilate, gripper_min / gripper_max or template_name. Needs Pillow; without it photos are sent as given. preprocess() sends the photo as given unless you pass resize="auto". Measured cost and gain (bytes, latency, COCO mAP): Clients › Python.

Prompts (serialized to the server's string format):

• box: [x, y, w, h] or a list of boxes → "x,y,w,h" joined by ;.
• point: [x, y] / [x, y, label] or a list (label 1=fg, 0=bg) → "x,y[,label]" joined by ;.
• prompt: free text, e.g. "cat. remote.". The client normalizes , and | to the GroundingDINO phrase separator . and appends a trailing . if missing.
• box_threshold / text_threshold: GroundingDINO score cutoffs (open-vocab models); see Open-vocab / Grounded-SAM.
• gripper_min / gripper_max: grasp models only — jaw-opening bounds in original-image pixels (e.g. gripper_min=20, gripper_max=150). Server filters out grasps outside the range.

Result types

Detection(bbox: list[float], cls: str, conf: float)       # bbox = [x, y, w, h], original px
Mask(rle: str, bbox: list[float], conf: float)
Classification(cls: str, conf: float)                      # top-K prediction
Grasp(
    x: float, y: float,       # grasp center in ORIGINAL image pixels
    theta: float,             # in-plane gripper-closing angle in radians
    width: float,             # jaw opening in ORIGINAL image pixels
    quality: float,           # analytic grasp score in [0, 1]
    cls: str = "",            # object class label ("" = class-agnostic)
    conf: float = 0.0,        # detector confidence (0.0 = class-agnostic)
)
Result(
    task, model, duration_ms,
    detections: list[Detection],
    masks: list[Mask],
    classifications: list[Classification],                 # task="classification"
    grasps: list[Grasp],                                   # task="grasp"
    depth_map: list[float] | None,                         # task="depth", row-major H×W
    depth_width: int | None,
    depth_height: int | None,
    embeddings: list[list[float]],                         # task="embedding" (CLIP)
)

Client(base64_arrays=True) asks the server for depth_map / embeddings as base64 float32 (exact values, several times cheaper for a large depth map). With numpy installed they then arrive as read-only, list-like FloatArray objects (numpy.asarray(x) without a copy; .tolist() before json.dumps or append). The default keeps plain lists.

Mask.to_ndarray(width, height) -> np.ndarray decodes the COCO-style column-major uncompressed RLE into a boolean (height, width) array (requires numpy). It is the exact inverse of the server's encoder.

Detection

# RF-DETR (COCO) or RT-DETR
res = c.predict("rf-detr", "photo.jpg")
for d in res.detections:
    print(d.cls, round(d.conf, 3), d.bbox)   # bbox = [x, y, w, h] in original px

Segmentation

# Box-prompted segmentation
from PIL import Image
res = c.predict("mobile-sam", "img.jpg", box=[50, 40, 120, 90])
w, h = Image.open("img.jpg").size
mask = res.masks[0].to_ndarray(width=w, height=h)   # bool (h, w) numpy array

# No-prompt → Automatic Mask Generator (segment everything, ~256 masks)
res = c.predict("mobile-sam", "img.jpg")
for m in res.masks:
    print(m.bbox, round(m.conf, 3))

# EfficientSAM and SAM2 — same prompt interface
res = c.predict("efficient-sam", "img.jpg", box=[50, 40, 120, 90])
res = c.predict("sam2", "img.jpg", box=[50, 40, 120, 90])

Background segmentation

background returns the support-surface (background) mask — the table or floor an object rests on — as a single segmentation mask. The result is 0 or 1 mask (no prompt needed), so always guard if res.masks:. The foreground (objects) is simply the complement of this mask: invert it client-side if that's what you need.

It picks an algorithm per request via method= (also the JSON/form field method):

from PIL import Image
W, H = Image.open("scene.jpg").size

res = c.predict("background", "scene.jpg")            # method=auto (depth→cv fallback)
if res.masks:                                          # 0 masks possible — always guard
    bg = res.masks[0].to_ndarray(width=W, height=H)   # support-surface (table/floor) mask
    fg = ~bg                                           # foreground = complement (invert)

# force a specific method / tune the sam+automask classification
res = c.predict("background", "scene.jpg", method="cv")
res = c.predict("background", "scene.jpg", method="automask", bg_max_area=60, grid_size=16)

Tuning knobs (areas are a percentage of the image area) apply only to the sam / automask methods — depth and cv ignore them:

Do NOT pass min_size / max_size to background. Those are an output bbox-area filter applied server-side after the model, and they will DROP the surface mask (its bbox spans the support surface). Use bg_max_area / fg_min_area instead. Also always guard if res.masks: — 0 masks is a valid result (e.g. method="depth" on a cluttered monocular scene with no clear plane, or thresholds set too aggressively).

Open-vocab / Grounded-SAM

# GroundingDINO — text → boxes
res = c.predict("grounding-dino", "img.jpg", prompt="cat. remote.")
for d in res.detections:
    print(d.cls, round(d.conf, 3), d.bbox)

# Grounded-SAM — text → boxes → masks
res = c.predict("grounded-sam", "img.jpg", prompt="cat. remote.")
print([d.cls for d in res.detections], "→", len(res.masks), "masks")

Tuning thresholds — each box is scored against every .-separated phrase and named after the best one, so a label is always a whole phrase ("canned coffee", never a fragment of it). The box is kept when that score is above box_threshold (default 0.3) and above text_threshold (default 0.25); with the defaults only box_threshold matters, and text_threshold changes the result only when you set it above box_threshold.

# box_threshold raises/lowers the detection cutoff (fewer/more boxes)
res = c.predict("grounding-dino", "coffee.jpg", prompt="canned coffee. cup.", box_threshold=0.35)

Both apply to grounding-dino, grounded-sam, grasp-gd and the GroundingDINO pass of the gdino-siglip* / rfdetr-gdino* models. None (default) defers to the model's manifest value. Every option, with real outputs: https://mtbui2010.github.io/vision_serve/clients/python/.

Depth estimation

import numpy as np
res = c.predict("depth-anything-v2", "img.jpg")
depth = np.array(res.depth_map).reshape(res.depth_height, res.depth_width)
# or MiDaS (faster, 256×256)
res = c.predict("midas", "img.jpg")

Classification

res = c.predict("efficientnet-b0", "img.jpg")
for cls_pred in res.classifications:
    print(cls_pred.cls, round(cls_pred.conf, 3))
# or MobileNetV3 (lighter)
res = c.predict("mobilenet-v3", "img.jpg")

CLIP embeddings

import numpy as np
res = c.predict("clip", "img.jpg")
vec = np.array(res.embeddings[0])     # shape (512,)
vec /= np.linalg.norm(vec)            # L2-normalize before cosine similarity

SCRFD / OCR

# Face detection — detections with cls="face" and 5 keypoints in the bbox extensions
res = c.predict("scrfd", "photo.jpg")
for d in res.detections:
    print(d.cls, round(d.conf, 3), d.bbox)

# OCR — text regions as detections; cls = recognized text string
res = c.predict("paddle-ocr", "doc.jpg")
for d in res.detections:
    print(d.cls, round(d.conf, 3), d.bbox)

Grasp detection

For planar grasp detection, results come back in grasps:

# Class-agnostic — whole-image automask → grasps
res = c.predict("grasp", "bin.jpg", gripper_min=20, gripper_max=150)
for g in res.grasps:
    print(f"q={g.quality:.3f}  x={g.x:.1f} y={g.y:.1f}  θ={g.theta:.3f}  w={g.width:.1f}")

# Class-aware — text-prompted detector → per-object grasps
res = c.predict("grasp-gd", "table.jpg", prompt="mug. bottle.",
                gripper_min=20, gripper_max=150)
for g in res.grasps:
    print(g.cls, round(g.quality, 3), g.contacts())  # contacts() → [[x0,y0],[x1,y1]]

# Pick the best grasp for a target class / pixel
from visionserve import select_target_grasp
target = select_target_grasp(res.grasps, cls="mug",
                              gripper_min=20, gripper_max=150)
if target:
    print("best grasp:", target.x, target.y, target.theta)

Fast single-target grasp (detect → select → grasp one box)

grasp-gd runs the analytic mask→grasp search on every detected object, which is wasteful when you only want to pick up one target. Instead, detect first, select the target client-side, then grasp just that box — the grasp model accepts a box prompt and segments + grasps only that region (one mask→grasp instead of N), so it scales with the target count, not the scene. With many objects this is several times faster.

from visionserve import Client, select_target_object, select_target_grasp
from PIL import Image

c = Client()
W, H = Image.open("scene.jpg").size

# 1) Boxes only (no masks) — grounding-dino is faster than grounded-sam here
det = c.predict("grounding-dino", "scene.jpg", prompt="mug. bottle.", text_threshold=0.15)

# 2) Select the target object client-side (by class, nearest pixel, largest, distance, …)
obj = select_target_object(det, cls="mug", near_point="center", image_size=(W, H))

# 3) Grasp ONLY that box → one segmentation + one mask→grasp
g = c.predict("grasp", "scene.jpg", box=obj.bbox, gripper_min=20, gripper_max=150)
best = select_target_grasp(g.grasps)
if best:
    print(best.pose)   # [x, y, width, theta] — robot-ready

The box prompt is honored by the class-agnostic grasp model (no detector). On grasp-gd the built-in GroundingDINO detector runs instead and the box is ignored — use the two-step flow above to target a single object.

See Grasp post-processing below for the full selection and filtering API.

Preprocessing parity — Client.preprocess() / Client.tokenize()

POST /api/preprocess returns exactly what a model is fed (no inference runs), so you can check the server prepares inputs the way your model was trained. A mismatch here never raises an error; it only makes the served model quietly worse.

x = c.preprocess("my-detector", "img.jpg")      # PreprocessResult (needs numpy)
x.inputs                                        # {onnx_input_name: np.ndarray}, bit-exact
x.meta                                          # {"orig_width", ..., "scale_x", "pad_x", ...} or None
np.abs(x.inputs["input"] - my_transform(pil)).max()
c.tokenize("siglip-text", "a photo of a cup")   # token ids, padded as served

Cameras and video — Client.watch()

watch() runs a model on a camera, a video file or a stream: a generator of one Result per processed frame. The server stays stateless (each frame is a normal predict()); the SDK reads the source on its own thread and keeps only the latest frame, so results stay live and memory bounded (a gap in res.frame_id = frames skipped). Every predict() option works per frame.

from visionserve import Client, open_source

c = Client()
for res in c.watch(0, "rf-detr", fps=5, track=True):          # webcam 0, <= 5 requests/s
    print(res.frame_id, [(d.cls, d.track_id) for d in res.detections])

# RGB-D: depth goes up only to models that read it (accepts_depth: `background`)
for res in c.watch("realsense", "background", method="depth", return_frames=True, max_frames=10):
    print(res.frame_id, len(res.masks), res.frame.depth.shape)

Options: fps (min. spacing of request starts), in_flight (concurrent requests, results in order), max_frames, duration, depth="auto"|"always"|"never", return_frames (res.frame: colour, aligned depth, depth_scale, intrinsics), track (simple client-side IoU tracker → Detection.track_id), drop_frames=False (send every frame: offline video). Depth on the wire: uint16 unchanged (0 = no reading), floats as float32 metres (NaN / inf / <= 0 → 0); a frame sent with depth is never resized. CLI: visionserve watch SOURCE --model M [--fps N] [--prompt ...] [--depth auto|always|never] [--max-frames N] [--track] [--json]. Full guide: website/docs/guides/camera-and-video.md.

Checkpoint converter — pip install visionserve[convert]

visionserve-convert / visionserve.convert.export() turn a PyTorch, RF-DETR, HuggingFace or (with [convert-tf], in its own environment) TensorFlow checkpoint into an installed VisionServe model. They verify it end to end: parity, preprocessing, outputs, optional mAP and speed. See deploy/README.md § "Bringing your own checkpoint".

Post-processing

All methods return a new Result; the original is not modified. They work on detections, masks, and classifications as appropriate.

from visionserve import Client, get_depth_at_detection

client = Client()
result = client.predict("rf-detr", "photo.jpg")

# Keep only high-confidence detections
result = result.filter_by_conf(min_conf=0.5)

# NMS to remove overlapping boxes
result = result.nms(iou_threshold=0.45)

# Top-5 predictions
result = result.top_k(5)

# Sort and group by class
result = result.sort_by_conf()
by_class = result.group_by_class()
for cls, r in by_class.items():
    print(f"{cls}: {len(r.detections)} detections")

# Combine depth model with detection. A depth MODEL's map is RELATIVE inverse depth in [0, 1]
# (larger = closer, no units) at the model's resolution: pass the original image size so the
# boxes are mapped onto it. For metres, use an RGB-D sensor array instead (see below).
depth = client.predict("midas", "photo.jpg")
depths = get_depth_at_detection(depth, result, image_size=(W, H))
for det, d in zip(result.detections, depths):
    print(f"{det.cls}: relative closeness={d:.2f}" if d is not None else f"{det.cls}: no depth")

get_depth_at_detection(depth_result, det_result, *, mode="median", depth_scale=None, image_size=None) (from visionserve.postprocess or the top-level visionserve package) returns List[Optional[float]] — one value per detection/mask, or None when the box has no valid depth. mode is "median" (default), "mean", "min" or "max".

• With a depth Result (midas, depth-anything-v2) the values are the server's RELATIVE inverse depth (min-max normalised per image, larger = closer, not metres); image_size=(W, H) of the original image is required, because the map is at the model's resolution, and depth_scale is refused.
• With a 2-D numpy array from an RGB-D sensor the values are in metres (uint16 is taken as millimetres unless depth_scale says otherwise); pass image_size if the array is not at the image's resolution.

Grasp post-processing

All grasp post-processing helpers are importable from visionserve or visionserve.postprocess.

Grasp fields and helpers

g = res.grasps[0]

# Robot-ready pose as a flat list
print(g.pose)              # [x, y, width, theta]  — for robot control

# Jaw-contact points
print(g.contacts())        # [[x0, y0], [x1, y1]]  — nested
print(g.contacts_flat())   # [x0, y0, x1, y1]      — flat, ready for ROS/serial

result.filter_grasps(max_per_object)

Limit the number of grasps per detected object — keeps the max_per_object highest-quality grasps of each object. A grasp counts for the detection it was planned on (it carries that detection's class and conf); only a class-agnostic grasp is grouped by the smallest detection/mask bbox containing its centre. Also available as a parameter to predict().

# Via predict() — applied immediately after the server response:
res = c.predict("grasp-gd", "bin.jpg", prompt="mug.", max_grasps_per_object=3)

# Or on an existing result:
res = res.filter_grasps(max_per_object=3)   # None keeps all

select_target_grasp()

Pick the single best grasp for execution. Candidates are first filtered by class and gripper-width feasibility, then scored on one or more criteria (each normalised to [0, 1]):

By default the most-specific available criterion is used (distance > near > quality). Pass weights={"quality": 0.5, "near": 0.5} for a weighted composite.

from visionserve import select_target_grasp, CameraIntrinsics

res = c.predict("grasp-gd", "bin.jpg", prompt="mug.", max_grasps_per_object=3)

# 1. By quality alone (default)
target = select_target_grasp(res.grasps)

# 2. Prefer grasps near a pixel (e.g. robot workspace centre)
target = select_target_grasp(res.grasps, target_point=(320, 240))

# 3. Prefer grasps at a specific 3D distance: needs METRIC depth from an RGB-D sensor,
#    aligned to the image. (A depth model's Result is relative and has no metres: it is refused.)
import numpy as np
depth_mm = np.asarray(realsense_depth, dtype=np.uint16)   # (H, W) millimetres
K = CameraIntrinsics(fx=600, fy=600, cx=320, cy=240)       # or [fx, fy, cx, cy]
target = select_target_grasp(
    res.grasps,
    target_distance=0.55,                 # metres
    depth_result=depth_mm,                # 2-D ndarray (uint16/float32) at image res
    intrinsics=K,
)
# Depth is normalised to METRES: integer arrays default to mm→m (×0.001), float
# arrays are taken as metres. Override with depth_scale=<m per unit>.

# 4. Filter by class + gripper bounds, weighted composite
target = select_target_grasp(
    res.grasps,
    cls="mug",
    gripper_min=20, gripper_max=150,
    target_point=(320, 240),
    weights={"quality": 0.4, "near": 0.6},
)

if target:
    print(target.pose)           # [x, y, width, theta]
    print(target.contacts_flat())# [x0, y0, x1, y1]

return_index=True returns (grasp_or_None, index) into the original list.

depth_result (a METRIC 2-D numpy array), intrinsics (CameraIntrinsics or [fx, fy, cx, cy]), and depth_scale are accepted the same way by select_target_object, grasp_distances and object_distances. These metric helpers refuse a depth-model Result (relative inverse depth); get_depth_at_detection accepts both (see above).

select_target_object()

For pipelines that need to pick ONE object first (e.g. "closest mug") before computing grasps, use select_target_object():

from visionserve import select_target_object

det = c.predict("rf-detr", "scene.jpg")
obj, idx = select_target_object(
    det, cls="cup",
    near_point="center",          # or (x, y) pixel
    image_size=(1280, 720),
    return_index=True,
)

Same scoring criteria as select_target_grasp but operates on Detection / Mask objects using conf, area, near, and distance.

grasp_distances() / object_distances()

True Euclidean camera→grasp (or camera→object) distance, computed by sampling the depth map at each grasp centre and back-projecting through the camera intrinsics.

from visionserve import grasp_distances, object_distances, CameraIntrinsics

K = CameraIntrinsics(fx=600, fy=600, cx=320, cy=240)
depth = c.predict("depth-anything-v2", "scene.jpg")

# Per-grasp distances (metres)
dists = grasp_distances(depth, res.grasps, K)
for g, d in zip(res.grasps, dists):
    print(f"grasp ({g.x:.0f},{g.y:.0f}) → {d:.2f}m" if d else "no depth")

# Per-object distances
odists = object_distances(depth, det, K)

Visualization with target grasp highlighted

from visionserve import draw, select_target_grasp

res = c.predict("grasp-gd", "bin.jpg", prompt="mug.", max_grasps_per_object=3)
target = select_target_grasp(res.grasps)

# All grasps drawn in quality colour; target drawn in red on top
annotated = draw(res, "bin.jpg", target_grasp=target)
annotated.save("out.png")

# Or via result.visualize():
res.visualize("bin.jpg", target_grasp=target).save("out.png")

Visualization with target box highlighted

Highlight a selected object box (e.g. from select_target_object) in red with a thicker outline via target_box. It accepts a Detection / Mask (matched by identity or bbox) or a raw [x, y, w, h]. If the box isn't one of the result's own detections/masks (e.g. it was selected on a separate detect call) it's drawn as a standalone red rectangle:

from visionserve import select_target_object

det = c.predict("grounding-dino", "scene.jpg", prompt="mug. bottle.")
obj = select_target_object(det, cls="mug", near_point="center", image_size=(W, H))

# Highlight the chosen object in red; all others in palette colours
det.visualize("scene.jpg", target_box=obj).save("out.png")

# Works with a raw bbox too (e.g. to mark the grasped region on a grasp result)
g = c.predict("grasp", "scene.jpg", box=obj.bbox)
g.visualize("scene.jpg", target_box=obj.bbox).save("out.png")

Size filtering — Result.filter_by_size()

Remove detections/masks whose bounding-box area is outside a range (client-side, on already-received results). The server's min_size / max_size fields do the same filtering server-side (as % of image area) before the response is sent.

# Absolute mode — area in pixels²
big = res.filter_by_size(min_size=5000)           # keep objects ≥ 5000 px²
small = res.filter_by_size(max_size=2000)          # keep objects ≤ 2000 px²
mid = res.filter_by_size(min_size=500, max_size=50000)

# Relative mode — fraction of image area (0.0–1.0), requires image dimensions
res_rel = res.filter_by_size(
    min_size=0.005,  # at least 0.5% of image area
    max_size=0.9,    # at most 90% of image area
    image_width=1280, image_height=720,
)

The method returns a new Result; the original is not modified.

Visualization — draw() / result.visualize()

Requires Pillow (pip install pillow or pip install 'visionserve[images]').

from visionserve import draw   # or: from visionserve.visualize import draw

# Works with any task — detection, segmentation, classification, depth
res = c.predict("rf-detr", "photo.jpg")
annotated = draw(res, "photo.jpg")   # → PIL.Image
annotated.save("out.jpg")

# Convenience method on Result:
res.visualize("photo.jpg").save("out.jpg")

# Control mask overlay opacity:
annotated = draw(res, "photo.jpg", alpha=0.6)

# Mask outlines, a title band, explicit sizes (default: scaled with the photo):
annotated = draw(res, "photo.jpg", mask_outline=True, title="rf-detr", font_size=20, line_width=3)

# Colours: one per class name (default) or one per item (the server's palette):
annotated = draw(res, "photo.jpg", color_by="index")

# Depth next to the photo with a far…near colour bar, or blended onto it:
draw(c.predict("midas", "photo.jpg"), "photo.jpg", depth="side").save("depth.jpg")

# Classifications as a bar panel on the right:
draw(c.predict("efficientnet-b0", "photo.jpg"), "photo.jpg", classes="bars")

# The prompts you send (dashed boxes, green dot = label 1, red cross = label 0):
from visionserve import draw_prompts
draw_prompts("photo.jpg", boxes=[50, 40, 120, 90], points=[[100, 80, 1], [20, 20, 0]]).save("p.jpg")

What gets drawn per task:

annotated = draw(res, "bin.jpg", max_grasps_per_object=3)

Examples

# RF-DETR / RT-DETR detection (optionally draw boxes):
python clients/python/examples/detect.py cat.jpg --model rf-detr --save out.png
python clients/python/examples/detect.py cat.jpg --model rt-detr --save out.png

# MobileSAM / EfficientSAM / SAM2 with a box prompt -> mask ndarray:
python clients/python/examples/segment.py img.jpg --model mobile-sam --box 50,40,120,90 --save mask.png
python clients/python/examples/segment.py img.jpg --model efficient-sam --box 50,40,120,90 --save mask.png
python clients/python/examples/segment.py img.jpg --model sam2 --box 50,40,120,90 --save mask.png

# Open-vocab (text prompt) — model must be available on the server:
python clients/python/examples/grounded.py img.jpg --prompt "cat. remote."

# Depth estimation:
python clients/python/examples/depth.py img.jpg --model depth-anything-v2 --save depth.png
python clients/python/examples/depth.py img.jpg --model midas --save depth.png

# Image classification:
python clients/python/examples/classify.py img.jpg --model efficientnet-b0
python clients/python/examples/classify.py img.jpg --model mobilenet-v3

Tests

The test suite is fully offline — it spins up a mock HTTP server in a thread and also round-trips the RLE codec against a reference port of the Go encoder. No running Go server is required.

# with pytest:
python -m pytest clients/python/tests -v

# or as a dependency-free self-test:
python clients/python/tests/test_client.py

Changelog

0.3.1

• Grasp grouping fix: Result.filter_grasps (so predict(max_grasps_per_object=) and draw) groups a grasp with the detection it was planned on, matched by the grasp's class and conf (within 1e-6; a tie is broken by the box containing the grasp centre). 0.3.0 used the smallest box containing the centre, so a bowl grasp lying inside the carrot's box took the carrot's place. Class-agnostic grasps keep the containing-box rule.
• Result.group_by_class(): each group holds only its class's grasps (a class-aware grasp by its class, a class-agnostic one by the detection box containing it, else ""); 0.3.0 copied every grasp into every group.
• draw() options: color_by="class" (new default: a stable colour per class name; "index" is the 0.3.0 look), mask_outline=, font_size= / line_width= (default: scaled with the photo's shorter side, unchanged at 640 × 426), depth="map"|"side"|"overlay"|"none" with a far … near colour bar, depth_colormap=, classes="text"|"bars"|"none", title=. Labels pick black or white text by contrast.
• draw_prompts(image, boxes=, points=, labels=): draws the prompts of a predict(box=, point=) call (dashed boxes, green dot = label 1, red cross = label 0).

0.3.0

• Client.watch(source, model, ...): run a model on a camera, video file or stream, one Result per processed frame (with frame_id, timestamp, and frame when return_frames=True). A reader thread keeps only the latest frame; fps spaces request starts, in_flight overlaps requests (results in order), max_frames / duration stop it, drop_frames=False sends every frame. A 503 is retried once after its Retry-After.
• visionserve.sources: one FrameSource interface (read(timeout) -> Frame | None, close()) and open_source(spec) over OpenCV (webcam / file / RTSP, with reconnect), GStreamer (gst-launch-1.0 subprocess, no PyGObject), Intel RealSense, Orbbec (SDK v2) and ROS 2 topics (no cv_bridge), plus IterSource for frames you already have. Vendor packages are optional extras, imported lazily: [opencv], [realsense], [orbbec].
• Depth policy: watch(depth="auto") uploads a frame's aligned depth only to models whose ModelInfo.accepts_depth is true (new in GET /api/models; Client.accepts_depth(model)); uint16 goes up unchanged, float depth as float32 metres with invalid pixels as 0.
• Tracking: watch(track=True) adds Detection.track_id with a small IoU tracker (visionserve.IoUTracker); Detection.to_json() includes it when set.
• Frame.depth_meters(), Frame.camera_intrinsics(); CLI visionserve watch; visionserve list --json prints accepts_depth.

0.2.0

• Client-side resizing, on by default (a user decision: it changes what is uploaded, so results on photos larger than a model's input move slightly; measured in the docs). Client(resize="auto", jpeg=True, jpeg_quality=90) shrinks a photo larger than the model can use to the server's per-model hint (GET /api/models: max_useful_side / max_useful_short_side), sends JPEG (4:4:4), and maps boxes, masks and grasps back to the original pixels; Result.client_resize records it with a reason. Only shrunk photos are re-encoded (JPEGs decoded at a reduced scale, Pillow's draft mode); with a loopback host only a shrink to half the sides or less is done. Mask.to_ndarray(original_w, original_h) keeps working on a shrunk photo's masks. resize="off" restores the 0.1.x uploads exactly. Also per call (predict(..., resize=, jpeg=, jpeg_quality=), preprocess(..., resize="auto")) and in the CLI (--resize auto|off|N, --no-jpeg, --jpeg-quality Q).
• ModelInfo.max_useful_side / max_useful_short_side; Client.useful_side(model); visionserve list --json prints both.