mlx-beam-vision

The towers that let mlx-beam see. Install it next to the engine (pip install mlx-beam[vision], or this package by name) and a checkpoint that ships a vision tower serves images: an image_url part with a data: URL in a chat request, the checkpoint's own processor renders the template, the tower encodes the image, and the engine's prefill takes the image's features in place of the placeholder tokens - beside text-only requests in the same batch, with the prefix cache keyed on the image's digest. Without this package the engine answers image input with a 400 that names the vision extra.

Towers vendored from mlx-vlm (MIT, see mlx_beam_vision/_vendor/VENDORED.md):

Not yet: audio and video input, quantized towers (the towers load at model precision; a package that ships a quantized tower is refused with a message). An image above 32 megapixels is refused before it is decoded.

The package installs torch and torchvision: the processors transformers ships for these families (the template rendering and the image-to-patches step) are written on them. The towers themselves run on MLX; torch does no inference here.

Every family is exercised in the tests on a tiny random tower against a direct forward; the real towers and processors are validated on Apple silicon with the checkpoints themselves.

/health.vision reports the family, the tower, the processor and the feature cache (encoder outputs by image digest, bounded in bytes).