xiaomi-lens-mnn

Run Xiaomi-Lens-0 (4.6 B vision-language grounding + segmentation model) on MNN with one command. No torch, no transformers, no 8.6 GB bf16 checkpoint — the CLI downloads three MNN graphs plus the embedding table and runs the vision encoder, the Qwen3.5 hybrid decoder and the pixel-mask decoder entirely in MNN (CPU).

uvx xiaomi-lens-mnn --image cow.jpg \
  --prompt 'a large brown cow standing in the grass'

Output:

[xiaomi-lens-mnn] fp16 result
  prompt tokens : 255 (216 image, grid 24x36)
  completion    : 32 tokens
  bboxes        : [[462, 302, 854, 754]]
  masks         : 1
  timings       : vision 1.80s, preprocess 0.04s, prefill 70.11s, decode 6.74s, mask 0.85s, total 79.54s
  overlay       : /path/lens-out/overlay.png

<bbox>[462, 302, 854, 754]</bbox><mask><|mask|></mask>

Files written to lens-out/: overlay.png (coloured masks on the input image), mask_0.png, masks.npy (float32 logits on the 384×576 model canvas, K×H×W) and result.json. result.json also records bboxes mapped back to the original image (0–1000), the raw bboxes_canvas, content_box and original_size.

Install

uvx runs it without installing anything globally:

# first run downloads ~10.5 GB (fp16) into ~/.cache/xiaomi-lens-mnn/fp16
uvx xiaomi-lens-mnn --image photo.jpg --prompt "the red car on the left"

# smaller / faster download (6.2 GB), slight mask degradation
uvx xiaomi-lens-mnn --image photo.jpg --precision int8

# ModelScope mirror (recommended from mainland China)
uvx xiaomi-lens-mnn --image photo.jpg --source ms

# fully offline after downloading the repo
uvx xiaomi-lens-mnn --image photo.jpg --model-dir ./Xiaomi-Lens-0-MNN-fp16

Classic install: pip install xiaomi-lens-mnn then xiaomi-lens-mnn --help. Requires Python 3.10–3.14 and a platform with an MNN wheel (macOS arm64/x86_64, Linux aarch64/x86_64, Windows x86_64).

If the package is not on your PyPI index yet, run the wheel attached to the model repo:

uvx --from https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-fp16/resolve/main/dist/xiaomi_lens_mnn-0.1.0-py3-none-any.whl \
  xiaomi-lens-mnn --image photo.jpg

# mainland-China mirror
uvx --from https://modelscope.cn/models/yunfeng/Xiaomi-Lens-0-MNN-int8/resolve/master/dist/xiaomi_lens_mnn-0.1.0-py3-none-any.whl \
  xiaomi-lens-mnn --image photo.jpg --precision int8 --source ms

CLI

xiaomi-lens-mnn --image IMAGE [--prompt PROMPT] [--precision fp16|int8]
                [--source auto|hf|ms] [--model-dir DIR] [--cache-dir DIR]
                [--threads N] [--max-new N] [--no-think] [--out-dir DIR] [--json]

Python API

from PIL import Image
from xiaomi_lens_mnn import LensMNN
from xiaomi_lens_mnn.download import ensure_model

model_dir = ensure_model("fp16", "auto")       # cached after the first call
runner = LensMNN(model_dir, threads=15)
result = runner.run(Image.open("cow.jpg"), "the large brown cow", out_dir="out")
print(result["bboxes"], result["masks"], result["timings"])

Speed

Measured on an Apple M5 Pro (48 GB), CPU backend, 15 threads, same 586×394 image, 255-token prompt (216 image tokens), 32 generated tokens, one mask. Lower is better. Full data: benchmarks.json in the weight repos.

The MNN decoder graph is incremental (one token per session run), so its prefill is a 255-step loop and is slower than torch's parallel prefill. Vision and mask are also a little slower in MNN than in torch/ONNX. The win is decode: MNN is 89× faster per token than the official HF CPU path, because torch's GatedDeltaNet conv1d falls back to the slow im2col kernel there. End to end, MNN fp16 is 7.9× and int8 8.6× faster than torch while using 44 % / 67 % less disk. int8 is ~8 % faster than fp16, with a small mask drift (see below).

ONNX Runtime sits between them: its fp32 decoder is numerically exact but pays a large MatMul cost on CPU; its fp16 ViT/mask graphs are fast (mask 0.15 s).

Precision / verification

int8 is weight-only 8-bit block quantization (--weightQuantBits 8 --weightQuantAsymmetric --weightQuantBlock 64). The ViT and mask decoder show visible numerical drift but the final binary mask is still within IoU 0.997. Use fp16 when mask boundaries matter.

How it works

Three MNN graphs:

1. lens_vit.mnn — Qwen3.5 ViT (24 blocks, 1024 hidden, patch 16, merge 2). Position-embedding interpolation and 2-D rotary cos/sin are computed in numpy outside the graph, so arbitrary patch grids work. Outputs pre-merge features (for the mask decoder) and merged image features (for the LLM).
2. lens_llm_decode.mnn — one-token Qwen3.5-4B decoder step with explicit KV / conv / recurrent states. 24 GatedDeltaNet layers + 8 full-attention layers, hidden 2560, vocab 248,078 (tied lm_head split into two 124,039-wide MatMuls). M-RoPE positions are computed in numpy with interleaved [11, 11, 10] sections. The <|mask|> hidden state is captured while decoding.
3. lens_mask.mnn — pixel-mask decoder over the 24×36 patch canvas: mask embedding, patch self-attention, channel reduce, two transposed convolutions + 4× bilinear upsampling, output 32 masks at 384×576.

The package itself has no torch/transformers dependency. It ports the Qwen2-VL smart-resize + patchify (BICUBIC, factor 32, min 65,536 / max 8,388,608 pixels), the official chat template, the tokenizer (tokenizers), M-RoPE and the vision position/rotary tables to numpy/Pillow. Parity with the official processor is byte-exact for token ids and < 1e-2 max for pixel values (fp16 rounding).

Weights

ModelScope mirror: yunfeng/Xiaomi-Lens-0-MNN-fp16 / ...-int8. The first run downloads manifest.json, the three graphs and the fp16 embedding table into ~/.cache/xiaomi-lens-mnn/<precision>/. Downloads resume after interruption. No token is required; public repos.

Limitations

• Fixed 576×384 input canvas (24×36 patches). Every image is fitted onto that canvas: near-3:2 images are resized directly, others are centre letterboxed with neutral grey and boxes/masks are mapped back to the original image. High-resolution detail is therefore reduced to ~0.22 MP; crop first if you need small-object accuracy. Re-export lens_mask for other canvas sizes.
• The decoder graph is incremental; prefill is sequential. See the speed table.
• CPU only (MNN Metal is not enabled; the release was verified on CPU).
• An image + one prompt per process. The states are reset for each run().
• int8 mask boundaries drift slightly (IoU 0.997 vs fp16); prefer fp16 for precise segmentation.
• The mask decoder is trained for the model's <|mask|> token; the CLI overlays only answer masks by default (thinking masks are counted but not drawn).

License

Apache-2.0, following the Xiaomi-Lens-0 release. The MNN graphs are derivative weights of SeerRay-Lab/Xiaomi-Lens-0.