mosaix 0.1.0

Maximum-throughput video inference for YOLO and any vision model, via batched gutter-padded mosaic tiling.

mosaix

Maximum-throughput video inference for YOLO — and any vision model.

mosaix turns an ordinary object detector into a firehose. On a single laptop RTX 4060 (8 GB) it pushes YOLO11n to ~1735 FPS end-to-end / 2178 FPS inference-only on real, multi-minute video — decode included — without exotic hardware, custom CUDA, or model retraining. Point it at a model and a video; it gives you per-frame detections and the throughput number.

import mosaix

pipe = mosaix.VideoPipeline.from_model("yolo11n.pt")
result = pipe.run("long_video.mp4")

print(f"{result.fps:.0f} FPS end-to-end, {result.total_detections} detections")
for frame in result:
    for det in frame.detections:
        print(frame.index, det.cls, det.conf, det.xyxy)

---

How it hits 1000+ FPS

A detector spends most of its time on per-image fixed overhead (launch, letterbox, NMS, host↔device copies), not on the pixels that matter. mosaix amortises that overhead across many frames at once:

                                 one GPU forward pass
        ┌───────────────────────────────────────────────────────────┐
 frames │  ┌────┬────┬────┐   ┌────┬────┬────┐        ┌────┬────┬────┐ │
 0..287 │  │ f0 │ f1 │ f2 │   │ f9 │f10 │f11 │  ...   │f279│f280│f281│ │
        │  ├────┼────┼────┤   ├────┼────┼────┤        ├────┼────┼────┤ │
        │  │ f3 │ f4 │ f5 │   │f12 │f13 │f14 │        │... │... │... │ │
        │  ├────┼────┼────┤   ├────┼────┼────┤        ├────┼────┼────┤ │
        │  │ f6 │ f7 │ f8 │   │f15 │f16 │f17 │        │... │...│f287 │ │
        │  └────┴────┴────┘   └────┴────┴────┘        └────┴────┴────┘ │
        │     mosaic 0           mosaic 1      ...        mosaic 31     │
        └───────────────────────────────────────────────────────────┘
                  grid = 9 cells       ×       batch = 32 mosaics
                          =  288 frames per single forward pass

1. Downscale every frame to a small cell (default 426×240, i.e. "240p").
2. Mosaic grid cells (default 9 → a 3×3 grid) into one image, with a blank gutter between cells.
3. Batch batch mosaics (default 32) into a single model forward pass — so each pass covers grid × batch = 288 frames.
4. Remap every detection back to the exact frame it came from.

Install

pip install mosaix                 # core (numpy + opencv)
pip install "mosaix[ultralytics]"  # + YOLO / RT-DETR support
pip install "mosaix[onnx]"         # + ONNX Runtime (GPU)
pip install "mosaix[all]"          # everything

GPU inference needs a CUDA-enabled PyTorch (for the Ultralytics backend) or onnxruntime-gpu (for the ONNX backend) — install those per your CUDA version.

---

Tested models & benchmarks

mosaix is built around bounding-box detectors, but the adapter layer also runs classifiers, taggers, depth, dense-pose and face models. The table below is measured on this repo's models so you know what to expect before you try one.

Test rig: NVIDIA RTX 4060 Laptop (8 GB), FP16 on CUDA, OpenCV decode (no decord). Clip: The Monkey Business Illusion — 854×480, ~30 fps, many people (a non-private, reproducible stand-in for crowded real footage). Each model ran in its own process (uncontested), sweeping configs 1×1 (no tiling), 4×32 and 9×32 (grid×batch). 600 frames per config for detectors, fewer for heavy whole-image nets.

These are relative, apples-to-apples numbers across families on a short 480p clip — they are decode-bound (note the low GPU-util), not peak. For tuned single-model peak (YOLO11n hits ~1735 e2e / 2178 infer FPS), see docs/performance.md. Install decord and use a 240p source to get there.

Throughput

e2e = decode→mosaic→infer→remap (what you get); infer = GPU forward only. tiling× = best e2e ÷ untiled (1×1) e2e — the speedup mosaicking buys. VRAM is the Torch allocator peak; ONNX/onnxruntime memory isn't visible to it (—).

Model	Task	Backend	CUDA e2e	CUDA infer	tiling×	best g×b	VRAM	CPU e2e
FastSAM-s	segment	ultralytics	308	494	5.5×	4×32	1187 MB	43
omniparser_icon_detect	detect (UI icons)	ultralytics	250	360	4.5×	4×32	684 MB	26
yolov12n-face	detect (face)	ultralytics	235	473	6.7×	9×32	413 MB	94
yolo11m	detect	ultralytics	231	355	4.3×	4×32	684 MB	23
yolo11m-pose	pose	ultralytics	223	349	4.7×	4×32	646 MB	21
rtdetr-l	detect (DETR)	ultralytics	223	271	8.5×	4×32	1268 MB	4.0
FastSAM-x	segment	ultralytics	220	265	4.7×	4×32	1803 MB	4.6
yolo11n-cls	classify	ultralytics	207	247	2.6×	1×32	123 MB	75
yolov12m-face	detect (face)	ultralytics	198	304	5.0×	4×32	644 MB	22
yolo11x-pose	pose	ultralytics	191	244	5.8×	4×32	1242 MB	4.7
yolov12l-face	detect (face)	ultralytics	190	268	8.0×	4×32	770 MB	16
yolo11m-seg	segment	ultralytics	189	268	4.4×	4×32	1186 MB	16
yolo11n	detect	ultralytics	186	309	3.0×	4×32	212 MB	94
yolo11x	detect	ultralytics	184	246	5.7×	4×32	1238 MB	5.2
yoloe-11s-seg	segment (open-vocab)	ultralytics	177	317	4.1×	4×32	652 MB	36
yolo11n-obb	oriented bbox	ultralytics	175	259	4.3×	4×32	212 MB	76
yolo11n-pose	pose	ultralytics	170	269	3.7×	4×32	213 MB	81
dw-ss_ucoco	pose (DWPose)	onnx¹	169	187	1.2×	1×32	—	154
yolo11n-seg	segment	ultralytics	155	243	3.6×	4×32	461 MB	71
320n	detect²	onnx	142	154	1.1×	4×32	—	146
depth_anything_v2_small	depth	transformers	122	132	2.1×	1×32	1169 MB	4.3
yolo11n.onnx	detect	onnx	102	110	1.3×	4×32	—	93
dw-mm_ucoco	pose (DWPose)	onnx¹	80	84	1.0×	1×32	—	77
insightface (buffalo_l)	detect (face, SCRFD)	insightface	70	73	1.1×	4×32	—	62
dw-ll_ucoco_384	pose (DWPose)	onnx¹	21	21	1.0×	1×1	—	21
sapiens_0.3b_goliath	body-part seg	torchscript¹	3.2	3.2	1.0×	1×1	2151 MB	timeout
yolox_l	detect	onnx³	2.6	2.6	1.0×	1×1	—	2.7
pixai	tagger (multi-label)	onnx	0.4	0.4	—	1×1	—	0.4
wd-eva02	tagger (multi-label)	onnx	0.4	0.4	—	1×1	—	0.4
densepose_r50_fpn	dense UV	torchscript	—	—	—	—	—	—
nlf_s / nlf_l	3D pose	torchscript	—	—	—	—	—	—

¹ Whole-image nets run one frame per input (no mosaic); tiling doesn't apply — batch is the only lever, so their speedup is modest. ² 320n loads as a detector but uses a non-standard 22-channel head (not COCO). ³ yolox_l is exported with a fixed batch of 1, so mosaix runs mosaics one-at-a-time — re-export with dynamic=True for real throughput. densepose/nlf load but expose no generic forward (they need their own project's inference code), so they're recognised but not benchmarkable here.

Reading it:

• Tiling pays off for detectors — 3–8.5× over untiled, peaking around grid=4 (4×32) on this decode-bound 480p clip. RT-DETR and the large/face models gain most.
• grid=4 won here, not grid=9 — because end-to-end is decode-bound (480p, OpenCV); the bigger 9×32 mosaic adds compute without feeding faster frames. On a 240p source with decord, grid=9/16 pull ahead (see performance docs).
• CPU is viable for nano models (yolo11n/-pose/-obb, yolov12n-face, 320n: 75–95 FPS) but collapses for large ones (RT-DETR, yolo11x, FastSAM-x: 4–5 FPS). Use CPU only for the n-class models.

Accuracy & the cost of tiling

Measured on coco128 (128 labelled COCO images, auto-downloaded, ~7 MB) with one mAP routine, native full-res vs the default 9×32 mosaic — so the drop is the tiling cost. Packing 9 frames into one 240p-cell mosaic shrinks small objects, so expect a real hit; it's smallest for big objects / larger models and worst for crowded tiny-object scenes (use grid=4 or bigger cells there).

Model	native mAP@.5	tiled (9×32) mAP@.5	retention	published (full COCO / source)
rtdetr-l	0.81	0.48	59 %	53.0 AP @.5:.95
yolo11x	0.71	0.41	58 %	54.7 mAP@.5:.95
yolo11m / -seg	0.66	0.38	58 %	51.5 mAP (32.0 mask)
yolo11n-seg	0.48	0.23	47 %	32.0 mAP mask
yolo11n	0.46	0.21	46 %	39.5 mAP@.5:.95
yolo11n.onnx	0.46	0.19	43 %	39.5 mAP@.5:.95

Published metrics for families coco128 can't score (different domains), from each model's authoritative source:

Family	Metric (published)
YOLO11 pose (n/m/x)	50.0 / 64.9 / 69.5 mAP-pose @.5:.95 (COCO)
YOLO11n-cls	70.0 % top-1 (ImageNet)
YOLO11n-obb	78.4 mAP@.5 (DOTAv1)
Depth-Anything-V2-Small	δ1 ≈ 0.724 (Sun-RGBD)
WD-EVA02 tagger	F1 ≈ 0.477 (Danbooru)
Sapiens-0.3B Goliath	mIoU 76.7 (body-part seg)
InsightFace buffalo_l (SCRFD-10GF)	WIDERFACE 95.2 / 93.9 / 83.1 (easy/med/hard)
YOLOX-l	49.7 AP @.5:.95 (COCO)

Reproduce everything:

python benchmarks/bench_all.py --models <models_dir> --video <clip.mp4> --devices cuda,cpu
python benchmarks/accuracy.py  --models <models_dir>     # coco128 native-vs-tiled mAP

Supported model families at a glance

Family	Examples here	How
YOLO detect / seg / pose / OBB / cls	yolo11{n,m,x}, -seg/-pose/-obb/-cls	UltralyticsAdapter (auto)
RT-DETR	rtdetr-l	auto (RTDETR)
YOLO-World / YOLOE / FastSAM / SAM	yoloe-11s-seg, FastSAM-{s,x}	auto
Face detection	yolov12{n,m,l}-face, InsightFace buffalo_{l,s}	auto / FaceAdapter
Generic YOLO ONNX	yolo11n.onnx, 320n, yolox_l	OnnxAdapter (v8/v5 auto)
Multi-label taggers	wd-eva02, pixai	TaggerAdapter (NCHW/NHWC)
Monocular depth (HF)	depth_anything_v2_small	DepthAdapter
Dense / pose nets (raw fwd)	sapiens, DWPose dw-*	TorchScriptAdapter
Loads, needs native API	densepose, nlf	recognised, not run here
Anything else	your model	CallableAdapter (see below)

Plug in any vision model

mosaix never talks to a model directly — it goes through a thin adapter. Three ways to bring a model, in increasing order of control:

1. By file — auto-detected backend

mosaix.VideoPipeline.from_model("yolo11n.pt")        # Ultralytics
mosaix.VideoPipeline.from_model("yolov8n.onnx")      # ONNX Runtime
mosaix.VideoPipeline.from_model("model.engine")      # TensorRT (via Ultralytics)

2. Any callable — the universal escape hatch

If your model is a Detectron2 net, a Transformers pipeline, a homemade detector — anything — wrap a function that maps a batch of mosaic images to boxes:

import numpy as np
from mosaix import VideoPipeline, TileConfig, InferenceConfig
from mosaix.adapters import CallableAdapter

def my_detector(mosaics):           # list[BGR uint8] -> list[(N,6)]
    out = []
    for img in mosaics:
        boxes = run_whatever(img)   # (N,6): x1,y1,x2,y2,conf,cls in mosaic pixels
        out.append(np.asarray(boxes, np.float32))
    return out

tile, infer = TileConfig(), InferenceConfig()
adapter = CallableAdapter(my_detector, tile, infer, name="my-model")
pipe = VideoPipeline(adapter)

The engine handles all the tiling/batching/remapping; your function only ever sees ordinary images and returns ordinary boxes. See examples/custom_model.py.

3. A custom adapter class

Subclass ModelAdapter, implement predict_batch, and register_adapter("name", Cls) to expose it everywhere. See docs/adapters.md.

---

Configuration

Everything is a documented dataclass. The defaults are tuned for an 8 GB GPU at 240p.

from mosaix import VideoPipeline, TileConfig, ReaderConfig, InferenceConfig

pipe = VideoPipeline.from_model(
    "yolo11n.pt",
    tile=TileConfig(
        grid=9,            # 9 = 3x3 mosaic; try 4 (2x2) for bigger objects
        batch=32,          # mosaics per forward pass; lower if you OOM
        cell_width=426,    # downscaled frame size
        cell_height=240,
        gutter=12,         # seam padding
    ),
    reader=ReaderConfig(
        stride=1,          # process every frame; 5 = 1-in-5
        threaded=True,     # overlap decode with GPU (essential for real FPS)
        backend="auto",    # "decord" if installed, else "opencv"
    ),
    inference=InferenceConfig(
        device="auto",
        half=True,         # FP16 — ~2x on modern GPUs
        conf=0.25,
        classes=[0],       # keep only COCO 'person'; None = all classes
    ),
)

Full reference: docs/configuration.md · Tuning guide: docs/performance.md.

---

Streaming (constant memory on long videos)

run() collects every result. For multi-hour videos, stream() yields one FrameResult at a time with bounded memory:

pipe = mosaix.VideoPipeline.from_model("yolo11n.pt")
for frame in pipe.stream("8_hour_stream.mp4"):
    process(frame)            # memory stays flat regardless of length
print(pipe.meter.summary())  # FPS, GPU mem, per-stage timing

---

Command line

mosaix bench yolo11n.pt video.mp4 --grid 9 --batch 32      # measure FPS
mosaix run   yolo11n.pt video.mp4 --out annotated.mp4 --classes 0
mosaix run   yolo11n.pt video.mp4 --jsonl dets.jsonl       # detections to JSONL
mosaix info  video.mp4                                     # probe metadata

Every config knob has a flag — run mosaix bench -h.

---

Why "true" throughput

Many benchmarks quote inference-only FPS with decode excluded. mosaix reports both: infer_fps (GPU forward passes only) and e2e_fps (decode + downscale + mosaic + inference + remap, what you actually get). The threaded reader overlaps decode with GPU so the end-to-end number stays close to the inference number on long videos — which is the only number that matters in production.

---

License

MIT — see LICENSE.