Osam

Run promptable vision models locally, on CPU, CUDA, WebGPU or CoreML.

Osam (/oʊˈsɑm/) is a tool to run open-source promptable vision models locally (inspired by Ollama).

Osam provides:

• Promptable Vision Models - Segment Anything Model (SAM), EfficientSAM, YOLO-World;
• Local APIs - CLI & Python & HTTP interface;
• Customization - Host custom vision models.

Installation

Pip

pip install osam[cpu]  # or osam[gpu] for NVIDIA GPUs

osam[gpu] installs onnxruntime-gpu together with the CUDA and cuDNN libraries, so no system CUDA install is needed. Install only one onnxruntime build: onnxruntime and onnxruntime-gpu overwrite each other's files.

For other GPUs (AMD, Intel, or NVIDIA without the CUDA libraries), install osam[webgpu]; osam then uses the GPU without --device. It runs on Vulkan, Direct3D 12 or Metal, and needs Python 3.11 or later. On Linux it also needs the system Vulkan loader (libvulkan.so.1).

For osam serve:

pip install osam[cpu,serve]

Quickstart

To run with EfficientSAM:

osam run efficientsam --image <image_file>

To run with YOLO-World:

osam run yoloworld --image <image_file>

Model library

Here are models that can be downloaded:

PS. sam, efficientsam is equivalent to sam:latest, efficientsam:latest.

Prompts

Speed

Median of 5 runs after a warm-up, excluding model loading.

Image encoding (ms)

One prompt (ms)

Point prompt; text for SAM3 and YOLO-World, which has no separate encoding.

Measured on examples/_images/dogs.jpg (2560x1600) with onnxruntime 1.23.2 (onnxruntime-gpu 1.31.1 for CUDA; onnxruntime 1.30.0 and onnxruntime-ep-webgpu 0.5.0 for WebGPU, on Vulkan and Metal). Run python examples/benchmark.py --device cpu --device cuda to measure on your machine. See docs/benchmark.md for SAM3 per prompt, load times and CPU/CoreML parity.

Custom download endpoint

Models are downloaded from their canonical URLs by default. To download from a mirror instead (for example, in networks where the canonical hosts are blocked), set OSAM_BLOB_ENDPOINT to a comma-separated list of endpoints tried in order:

# Try the mirror first, then fall back to the canonical host.
export OSAM_BLOB_ENDPOINT="https://mirror.example.com,direct"

Each blob is fetched from <endpoint>/<sha256-hex>; the reserved value direct uses the model's canonical URL. Downloads are verified by SHA-256 regardless of the endpoint, so a mirror cannot serve corrupted data. When unset, only direct is used. Omit direct from the list (for example OSAM_BLOB_ENDPOINT="https://mirror.example.com") to disable the canonical fallback and serve every blob from the mirror only.

Execution providers

Pick a device per model with --device on the CLI or device= in Python:

osam run sam3 --image examples/_images/dogs.jpg --text dog --device coreml

model = osam.apis.get_model_type_by_name("sam3")(device="cuda")

cpu, cuda, coreml and webgpu are supported. An unavailable device raises an error instead of falling back to CPU. With coreml, SAM3 runs its point decoder on CPU and the rest on CoreML. On Apple silicon, SAM3 encodes an image faster on CoreML than on WebGPU, but its first CoreML load takes about 40 s.

Without a device, inference runs on CUDA when the installed onnxruntime has it, then on WebGPU when osam[webgpu] finds a GPU, otherwise on CPU. A session that fails to start on the GPU, for example without a Vulkan driver, runs on CPU. model.device tells which one was picked. To use another accelerator (CoreML on macOS, DirectML on Windows, TensorRT, ROCm), set OSAM_ONNX_PROVIDERS to a comma-separated list tried in the given order:

export OSAM_ONNX_PROVIDERS="CoreMLExecutionProvider,CPUExecutionProvider"

CoreML ships in the osam[cpu] build on macOS. For DirectML and other builds, install osam without extras plus the matching onnxruntime package (for example onnxruntime-directml). If a session cannot be created with the given list, osam logs the error and falls back to CPUExecutionProvider.

Usage

CLI

# Run a model with an image
osam run efficientsam --image examples/_images/dogs.jpg > output.png

# Get a JSON output
osam run efficientsam --image examples/_images/dogs.jpg --json
# {"model": "efficientsam", "mask": "..."}

# Give a prompt
# 1. point prompt (x,y[,label]: foreground=1 by default, background=0)
osam run efficientsam --image examples/_images/dogs.jpg \
  --point 1439,504 --point 1439,1289 > efficientsam_point.png
# 2. box prompt with sam2 (x1,y1,x2,y2)
osam run sam2 --image examples/_images/dogs.jpg \
  --box 1233,376,1649,691 > sam2_box.png
# 3. text prompt
osam run sam3 --image examples/_images/dogs.jpg --text dog > sam3_text.png
# 4. box prompt with sam3: one object without --text, as with sam2, or every
# object like the example box with --text visual
osam run sam3 --image examples/_images/dogs.jpg \
  --text visual --box 1233,376,1649,691 > sam3_box.png
# 5. text prompt with sam3, excluding a region
osam run sam3 --image examples/_images/dogs.jpg \
  --text dog --negative-box 1200,400,1700,1400 > sam3_negative_box.png

# Or give the whole prompt as JSON (point labels: background=0, foreground=1,
# box=2,3, negative box=4,5)
osam run sam2 --image examples/_images/dogs.jpg \
  --prompt '{"points": [[1233, 376], [1649, 691]], "point_labels": [2, 3]}'
# from a file (@prompt.json) or stdin (-), with a threshold on top
osam run sam3 --image examples/_images/dogs.jpg --prompt @prompt.json \
  --score-threshold 0.5

Output images: 'efficientsam_point.png', 'sam2_box.png', 'sam3_text.png', 'sam3_box.png'

Python

import numpy as np
from PIL import Image

import osam

image = np.asarray(Image.open("examples/_images/dogs.jpg"))
request = osam.types.GenerateRequest(
    model="efficientsam",
    image=image,
    prompt=osam.types.Prompt(points=[[1439, 504], [1439, 1289]], point_labels=[1, 1]),
)
response = osam.apis.generate(request=request)
annotation = response.annotations[0]
mask = np.zeros(image.shape[:2], dtype=np.bool_)
bbox = annotation.bounding_box
mask[bbox.ymin:bbox.ymax + 1, bbox.xmin:bbox.xmax + 1] = annotation.mask
Image.fromarray(mask).save("mask.png")

Input and output images ('dogs.jpg', 'mask.png').

HTTP

# pip install osam[cpu,serve]  # required for `osam serve`

# Get up the server
osam serve

# POST request
base64 < examples/_images/dogs.jpg \
  | jq -Rs '{model: "efficientsam", image: gsub("\n"; "")}' \
  | curl 127.0.0.1:11368/api/generate -X POST --json @- \
  | jq -r '.annotations[0].mask' | base64 --decode > mask.png