xiaomi-lens-mnn

Run Xiaomi-Lens-0 (4.6 B vision-language grounding + segmentation model) on MNN with one command. No torch, no transformers, no 8.6 GB bf16 checkpoint — the CLI downloads three MNN graphs plus the embedding table and runs the vision encoder, the Qwen3.5 hybrid decoder and the pixel-mask decoder entirely in MNN (CPU). Choose fp16 (10.5 GB), int8 (6.2 GB) or int4 (4.1 GB, 4-bit decoder + 8-bit vision).

uvx xiaomi-lens-mnn --image cow.jpg \
  --describe 'a large brown cow standing in the grass'

Output:

[xiaomi-lens-mnn] fp16 result
  prompt tokens : 255 (216 image, grid 24x36)
  completion    : 32 tokens
  bboxes        : [[462, 302, 854, 754]]
  masks         : 1
  timings       : vision 1.80s, preprocess 0.04s, prefill 70.11s, decode 6.74s, mask 0.85s, total 79.54s
  overlay       : /path/lens-out/overlay.png

<bbox>[462, 302, 854, 754]</bbox><mask><|mask|></mask>

Prompting

--describe EXPR builds the instruction the model expects for grounding:

Locate all instances in the image that match the description "EXPR". Report bbox coordinates and masks.

Pass the same text to --prompt if you prefer (or if you are on 0.1.0). The instruction shape matters: a bare phrase such as --prompt 'a large brown cow' is answered as a caption and comes back without a bbox or mask. Xiaomi-Lens-0 also answers free-form questions, so --prompt itself is left unmodified — --describe is the safe way to ask for boxes and masks.

Files written to lens-out/: overlay.png (coloured masks on the input image), mask_0.png, masks.npy (float32 logits on the 384×576 model canvas, K×H×W) and result.json. result.json also records bboxes mapped back to the original image (0–1000), the raw bboxes_canvas, content_box and original_size.

Install

uvx runs it without installing anything globally:

# first run downloads ~10.5 GB (fp16) into ~/.cache/xiaomi-lens-mnn/fp16
uvx xiaomi-lens-mnn --image photo.jpg --describe "the red car on the left"

# smaller / faster download (6.2 GB), slight mask degradation
uvx xiaomi-lens-mnn --image photo.jpg --precision int8 \
  --describe "the red car on the left"

# smallest download (4.1 GB), 4-bit decoder, int8 vision
uvx xiaomi-lens-mnn --image photo.jpg --precision int4 \
  --describe "the red car on the left"

# ModelScope mirror (recommended from mainland China)
uvx xiaomi-lens-mnn --image photo.jpg --source ms --describe "the red car on the left"

# fully offline after downloading the repo
uvx xiaomi-lens-mnn --image photo.jpg --model-dir ./Xiaomi-Lens-0-MNN-fp16 \
  --describe "the red car on the left"

Classic install: pip install xiaomi-lens-mnn then xiaomi-lens-mnn --help. Requires Python 3.10–3.14 and a platform with an MNN wheel (macOS arm64/x86_64, Linux aarch64/x86_64, Windows x86_64).

CLI

xiaomi-lens-mnn --image IMAGE [--describe EXPR | --prompt PROMPT]
                [--precision fp16|int8|int4] [--source auto|hf|ms] [--model-dir DIR]
                [--cache-dir DIR] [--threads N] [--max-new N] [--no-think]
                [--out-dir DIR] [--json]

flag	default	meaning
--image	required	input image, any format Pillow reads
--describe	—	build the grounding instruction for this expression (bbox + mask)
--prompt	cow grounding demo	free-form instruction or question, passed through unchanged
--precision	fp16	fp16, int8, or int4 (4-bit decoder + int8 ViT/mask)
--source	auto	hf, ms, or auto (try HF, fall back to ModelScope)
--model-dir	—	already downloaded repo directory; skips the network
--cache-dir	~/.cache/xiaomi-lens-mnn	download cache
--threads	15	MNN CPU threads
--max-new	128	maximum generated tokens
--no-think	off	disable the <think> block (shorter, faster, less reasoning)
--out-dir	lens-out	output directory
--json	off	print the full result JSON

Python API

from PIL import Image
from xiaomi_lens_mnn import LensMNN, grounding_prompt
from xiaomi_lens_mnn.download import ensure_model

model_dir = ensure_model("fp16", "auto")       # cached after the first call
runner = LensMNN(model_dir, threads=15)
result = runner.run(
    Image.open("cow.jpg"), grounding_prompt("the large brown cow"), out_dir="out")
print(result["bboxes"], result["masks"], result["timings"])

grounding_prompt("...") is the same expansion as --describe; passing a bare phrase to runner.run returns a caption instead of a mask.

Speed

Measured on an Apple M5 Pro (48 GB), CPU backend, 15 threads, same 586×394 image, 255-token prompt (216 image tokens), 32 generated tokens, one mask. Lower is better. Full data: benchmarks.json in the weight repos.

stage	torch fp32 (HF)	ONNX Runtime	MNN fp16	MNN int8
vision (24×36 grid)	0.74 s	1.41 s	1.80 s	1.84 s
LLM prefill (255 tok)	24.78 s	84.46 s	70.11 s	64.46 s
LLM decode (32 tok)	600.07 s	10.36 s	6.74 s	6.10 s
pixel mask (1 mask)	0.35 s	0.15 s	0.85 s	0.53 s
end to end	625.9 s	96.4 s	79.5 s	73.0 s
speed-up vs torch	1×	6.5×	7.9×	8.6×
weights on disk	18.5 GB (fp32)	17.6 GB	10.5 GB	6.2 GB
decode per token	18.75 s	0.324 s	0.211 s	0.191 s

The MNN decoder graph is incremental (one token per session run), so its prefill is a 255-step loop and is slower than torch's parallel prefill. Vision and mask are also a little slower in MNN than in torch/ONNX. The win is decode: MNN is 89× faster per token than the official HF CPU path, because torch's GatedDeltaNet conv1d falls back to the slow im2col kernel there. End to end, MNN fp16 is 7.9× and int8 8.6× faster than torch while using 44 % / 67 % less disk. int8 is ~8 % faster than fp16, with a small mask drift (see below).

ONNX Runtime sits between them: its fp32 decoder is numerically exact but pays a large MatMul cost on CPU; its fp16 ViT/mask graphs are fast (mask 0.15 s).

MNN precision head-to-head

All three MNN builds re-measured back-to-back in one idle session (same machine, image, prompt, 32 generated tokens, one mask). Absolute times drift 10–20 % between sessions with thermal state; compare within this table, not against the older one above.

stage	fp16	int8	int4
vision	1.55 s	1.34 s	1.34 s
LLM prefill (255 tok)	51.21 s	47.45 s	46.61 s
LLM decode (32 tok)	4.81 s	4.85 s	4.63 s
pixel mask	0.51 s	0.37 s	0.34 s
end to end	58.1 s	54.0 s	52.9 s
decode per token	0.150 s	0.152 s	0.145 s
weights on disk	10.5 GB	6.2 GB	4.1 GB

int4 is ~2 % faster than int8 and ~9 % faster than fp16. Decode is nearly identical for int8/int4: the 4-bit kernels save memory bandwidth but pay it back in dequantization. The real win is download size — 4.1 GB instead of 6.2 GB.

Precision / verification

check	fp16	int8	int4
generated text / bbox vs HF	identical	identical	bbox digits shift ≤ 1.6 % of image width
pixel-mask IoU vs HF (cow)	1.0000	0.9970	0.9964
5-image mask IoU vs fp16, mean / worst	—	0.9972 / 0.9945	0.9863 / 0.9546
5-image bbox IoU vs fp16, mean / worst	—	1.000 / 1.000	0.979 / 0.936
peak RSS during a 64-token run	18.4 GB	19.1 GB	20.2 GB
decoder logits max / mean abs error (8 steps)	1.8e-3 / 3.6e-5	0.4268 / 0.3519	not measured
ViT merged max / mean abs error	7.2e-3 / 3.6e-5	1.095 / 2.8e-3	same as int8
mask logits max / mean abs error	6.0e-3 / 2.2e-3	3.143 / 0.2852	same as int8

int8 is weight-only 8-bit block quantization (--weightQuantBits 8 --weightQuantAsymmetric --weightQuantBlock 64). The ViT and mask decoder show visible numerical drift but the final binary mask is still within IoU 0.997.

int4 is mixed precision: the 4.2 B decoder uses 4-bit block-wise asymmetric weights (--weightQuantBits 4 --weightQuantAsymmetric --weightQuantBlock 64), while the ViT and mask decoder stay int8. A pure 4-bit build saves only ~0.2 GB more but degrades masks: 5-image mean IoU 0.906, worst 0.689 (Mona Lisa). 4-bit ViT alone was the main culprit (0.841 on that image at block 64, 0.969 with block 32); keeping vision in 8 bits is the better trade.

int4 reduces download/disk, not runtime RAM. MNN's CPU backend materializes weights into fp32 working copies, so a 4-bit decoder expands ~8× in RAM and the measured peak is slightly higher than int8 (20.2 vs 19.1 GB), not lower.

Use fp16 when mask boundaries must be exact, int4 when the 4.1 GB download or repo size matters.

How it works

Three MNN graphs:

1. lens_vit.mnn — Qwen3.5 ViT (24 blocks, 1024 hidden, patch 16, merge 2). Position-embedding interpolation and 2-D rotary cos/sin are computed in numpy outside the graph, so arbitrary patch grids work. Outputs pre-merge features (for the mask decoder) and merged image features (for the LLM).
2. lens_llm_decode.mnn — one-token Qwen3.5-4B decoder step with explicit KV / conv / recurrent states. 24 GatedDeltaNet layers + 8 full-attention layers, hidden 2560, vocab 248,078 (tied lm_head split into two 124,039-wide MatMuls). M-RoPE positions are computed in numpy with interleaved [11, 11, 10] sections. The <|mask|> hidden state is captured while decoding.
3. lens_mask.mnn — pixel-mask decoder over the 24×36 patch canvas: mask embedding, patch self-attention, channel reduce, two transposed convolutions + 4× bilinear upsampling, output 32 masks at 384×576.

The package itself has no torch/transformers dependency. It ports the Qwen2-VL smart-resize + patchify (BICUBIC, factor 32, min 65,536 / max 8,388,608 pixels), the official chat template, the tokenizer (tokenizers), M-RoPE and the vision position/rotary tables to numpy/Pillow. Parity with the official processor is byte-exact for token ids and < 1e-2 max for pixel values (fp16 rounding).

Weights

repo	precision	files
Xiaomi-Lens-0-MNN-fp16	fp16	ViT 633 MB, mask 159 MB, decoder 7.8 GB, embeddings 1.2 GB
Xiaomi-Lens-0-MNN-int8	int8	ViT 337 MB, mask 85 MB, decoder 4.2 GB, embeddings 1.2 GB
Xiaomi-Lens-0-MNN-int4	int4 decoder + int8 vision	ViT 337 MB, mask 85 MB, decoder 2.4 GB, embeddings 1.2 GB
Xiaomi-Lens-0-onnx	fp32 / fp16	ONNX graphs for re-conversion

ModelScope mirror: yunfeng/Xiaomi-Lens-0-MNN-fp16 / ...-int8 / ...-int4. The first run downloads manifest.json, the three graphs and the fp16 embedding table into ~/.cache/xiaomi-lens-mnn/<precision>/. Downloads resume after interruption. No token is required; public repos.

Limitations

• Fixed 576×384 input canvas (24×36 patches). Every image is fitted onto that canvas: near-3:2 images are resized directly, others are centre letterboxed with neutral grey and boxes/masks are mapped back to the original image. High-resolution detail is therefore reduced to ~0.22 MP; crop first if you need small-object accuracy. Re-export lens_mask for other canvas sizes.
• The decoder graph is incremental; prefill is sequential. See the speed table.
• CPU only (MNN Metal is not enabled; the release was verified on CPU).
• An image + one prompt per process. The states are reset for each run().
• Grounding needs an instruction that asks for bbox + mask; use --describe (or grounding_prompt) rather than a bare object name.
• int4 does not lower peak RAM (MNN expands quantized weights to fp32); it lowers download and disk only. Peak measured 20.2 GB vs 19.1 GB for int8.
• int8 mask boundaries drift slightly (IoU 0.997 vs fp16); int4 drifts more on some images (worst 0.955 over five demo images). Prefer fp16 for precise segmentation.
• The mask decoder is trained for the model's <|mask|> token; the CLI overlays only answer masks by default (thinking masks are counted but not drawn).

License

Apache-2.0, following the Xiaomi-Lens-0 release. The MNN graphs are derivative weights of SeerRay-Lab/Xiaomi-Lens-0.