qwen-image-mnn

Run Qwen-Image-2.1 text-to-image on-device with the MNN weights — no torch, no diffusers, no 33 GB checkpoint. One command through uvx:

uvx qwen-image-mnn "a red panda riding a bicycle" --steps 40 --out panda.png

The CLI downloads the converted MNN weights on first use (~30 GB for --quant fp16) into ~/.cache/qwen-image-mnn/<quant>, then renders offline. The official pipeline defaults are used unless you say otherwise: 40 steps, 1024 px, true_cfg_scale=1.0.

What runs where

int8/int4 are weight-only quantizations (mnnconvert --weightQuantBits … --weightQuantBlock 32 --hqq) of the same graphs; the VAE stays fp32 at every level. int8 stays inside the model's own dtype noise — the fp16-vs-bf16 reference run drifts 6.3e-3 → 2.0e-1 while int8 drifts 1.4e-3 → 2.4e-2, image PSNR 52 dB against the reference render (fp16: 43.6 dB). int4 is measurably worse (4.4e-1, PSNR 28.5 dB): use it when 7 GB matters more than the difference.

Every graph is exported from the released diffusers checkpoint to ONNX and then converted with mnnconvert. The tokenizer, the flow-match scheduler and all the joint-sequence bookkeeping (3-axis RoPE, block-causal mask, modulation rows) are plain numpy in this package, because they are cheap — the transformer itself is the only part that needs MNN.

Install / run

uvx qwen-image-mnn --help              # from PyPI
uvx --from /path/to/qwen-image-mnn qwen-image-mnn "a photo of a cat" --seed 7   # or a checkout

Options that matter:

--quant fp16           weight precision to download: fp16 | int8 | int4
--size 1024            square side in pixels (or --height/--width)
--steps 40             denoising steps (40 is the released pipeline's default)
--seed 0               seed for the initial latents
--true-cfg-scale 1.0   >1 with --negative-prompt enables classifier-free guidance
--backend cpu          the only backend the weights were verified on; metal/opencl are
                       experimental on MNN 3.6.1/macOS
--precision high       keep this: `low` swaps the CPU kernels to fp16 *arithmetic*, which
                       overflows the 32-block residual stream (absmax 2.6e36, output full of
                       inf). `high` and `normal` are bit-identical, so there is nothing to gain.
--model-dir DIR        use a local converted-model directory instead of downloading
--latents X.npy        start from fixed packed latents (reproducible comparisons)
--dump-latents X.npy   save every step's latents
--prompt-embeds X.npy  supply prompt embeddings (e.g. saved from the reference pipeline) and
                       skip the text encoder -- this isolates the DiT + VAE when comparing

Speed

CPU only (Metal numbers below), precision=high, 10 threads, on this 15-core Mac. These are warm numbers — the minimum of repeated calls inside one process, which is what a multi-step render actually pays:

Weight-only quantization buys size, not speed: the weights are dequantized as they are paged in, so every precision runs the same multiply-accumulates and the three columns land inside noise of each other. What a cold process pays on top is MNN's per-process dynamic-shape planning: ~23 s to resize the DiT session, so its first step is ~35 s instead of ~13 s of actual work, and ~20–30 s for the text encoder's first prompt instead of 0.4 s. Quantization does not remove that either, though it does dent it a little on the smaller graphs (int8: 22 s against 30 s for the encoder's first prompt, 32 s against 37 s for the DiT's first step).

A full 40-step render at the official 1024 px has been clocked at 4817 s (80 min) on this machine while it was carrying other work — but the same step measured 58 s warm a few hours later on the same (still busy) machine, which puts the same render nearer 40 min. Per-step time tracks how much else the machine is doing, so read every number here as a ballpark, not a benchmark. The step is attention-dominated at 1024 px (attention is 51 s of the 58 s against 3.3 s of 10.1 s at 512 px), so it is the sequence length, not the weights, that costs.

How the numbers were verified

Each graph was run against its source with identical inputs:

End to end, with both sides starting from the same latents: the MNN pipeline's latents drift 5.0e-3 (step 1) … 3.1e-2 (step 8) relative to the released pipeline's own fp16 run, image PSNR 43.6 dB. The released model compared against itself across dtypes (fp16 vs bf16, same seed and latents) drifts up to 2.0e-1 in the same job — the conversion is closer to the model than the model is to itself once the dtype moves. tools/ref_noise_floor.sh reproduces that floor, tools/compare_e2e.py is the comparison and its tolerance is sized from it.

The fp32 VAE is what the published numbers were measured with; the fp16 VAE variants kept in tools/ are for the record only (the decoder holds PSNR 43 dB, the encoder's mean error grows to 9e-3).

Text-only conversion notes, the text-encoder bisect (block-level taps that found the MNN importer bug) and the fixups that rewrite the opset-18 dynamo output into the opset-17 shape MNN's importer actually reads correctly live in tools/ (this package's sibling repo): export_dit_onnx.py, export_text_encoder_onnx.py, export_vae_onnx.py, onnx_fixups.py.

Current scope

Text-to-image (T2I) on the CPU backend. Image editing (the "创改一体" path) needs the vision tower and the deepstack merge, which is not part of these graphs yet.

Metal on macOS 26 with MNN 3.6.1 miscomputes: the Metal tensor-API capability probe fails to compile against the new MetalPerformancePrimitives headers and the fallback kernels return garbage. MNN master fixes the probe, and a locally built master library computes all three graphs correctly on Metal (VAE max rel 1.5e-3, DiT 2.2e-3) — but only the VAE gets faster (3.1x); the DiT is ~4x slower than CPU in the same probe, so CPU is the fastest full path even once an upstream release ships the fix. The pip wheel cannot load a locally built libMNN.dylib either, so none of this is reachable from this package today.

中文速览

uvx qwen-image-mnn "提示词" --steps 40 --out out.png 即可出图：首次运行按 --quant 把权 重下载到 ~/.cache/qwen-image-mnn/<quant>（fp16 30.7 GB / int8 21.4 GB / int4 14.4 GB），之后 完全离线。默认参数与官方一致：40 步、1024 px。pip 版 MNN 上请用默认 --backend cpu --precision high：low 精度会溢出，Metal 在 MNN 3.6.1 结果异常（上游 master 已修探针，但实测 只有 VAE 变快 3.1 倍，DiT 反而比 CPU 慢 4 倍，且 pip 包无法加载自编译库）。int8 与 fp16 同处于 模型自身 dtype 噪声之内，推荐替代；int4 有可测差异（PSNR 28.5 dB vs 52.0 dB），仅在体积敏感时 使用。量化只省体积不省时间：三档稳态步时相同（512 px 约 10.1/10.4/10.4 s，1024 px 约 58 s/步，且 1024 px 下耗时几乎全在注意力上），进程内首次调用还要一次性付 MNN 的动态 shape 规划（DiT 约 23 s resizeSession + 35 s 首步），详见 Speed 一节。当前覆盖文生图；图生图/图像编辑还需要视觉塔与 deepstack 融合，尚未包含。