mlx-vlm-batch-outlines 0.1.0

mlx-vlm-batch-outlines

mlx-vlm-batch-outlines is a small Qwen-focused MLX vision-language package for:

• multimodal chat with images
• batched multimodal chat
• constrained decoding with llguidance
• structured outputs from Pydantic, regex, CFG, or JSON Schema
• simple video inference helpers

It is intentionally narrow. It is built for local MLX/Qwen workflows and does not try to support every mlx-vlm backend or the full Outlines API surface.

Attribution

This project heavily reuses and adapts ideas and code paths from:

• mlx-vlm
• outlines

In particular:

• the MLX/Qwen multimodal runtime and model code are derived from mlx-vlm
• the constrained decoding architecture and structured-output interface are derived from outlines

Scope

This package currently targets Qwen vision models exposed through MLX, such as:

• mlx-community/Qwen3.5-4B-MLX-4bit
• mlx-community/Qwen2-VL-2B-Instruct-4bit

The public API is:

• load(...)
• chat(...)
• chat_stream(...)
• batch_chat(...)
• video_chat(...)
• video_chunk_process(...)

Structured output helpers:

• Regex
• CFG
• JsonSchema
• regex(...)
• cfg(...)
• json_schema(...)

Install

uv sync

Or with plain pip:

pip install -e .

Quick Start

from PIL import Image
from mlx_vlm_batch_outlines import chat, load

model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")
image = Image.open("cat.jpeg")

result = chat(
    model,
    processor,
    [
        {"role": "system", "content": "Answer concisely."},
        {"role": "user", "content": ["Describe this image.", image]},
    ],
    max_tokens=80,
)

print(result.text)

Batched Image Chat

from PIL import Image
from mlx_vlm_batch_outlines import batch_chat, load

model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")
cat = Image.open("cat.jpeg")
dog = Image.open("dog.jpeg")

results = batch_chat(
    model,
    processor,
    [
        [{"role": "user", "content": ["Describe this image.", cat]}],
        [{"role": "user", "content": ["Describe this image.", dog]}],
    ],
    max_tokens=80,
)

for text in results.texts:
    print(text)

Structured Outputs

Pydantic

from PIL import Image
from pydantic import BaseModel
from mlx_vlm_batch_outlines import chat, load


class VisualSummary(BaseModel):
    primary_subject: list[str]
    subject_count: int
    setting: str
    short_description: str


model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")
image = Image.open("cat.jpeg")

result = chat(
    model,
    processor,
    [
        {
            "role": "user",
            "content": ["Describe this image as JSON using the requested schema.", image],
        }
    ],
    output_type=VisualSummary,
    max_tokens=140,
)

print(result.model_dump())

CFG

from PIL import Image
from mlx_vlm_batch_outlines import CFG, chat, load

model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")
image = Image.open("cat.jpeg")

result = chat(
    model,
    processor,
    [
        {
            "role": "user",
            "content": [
                "What animal is most prominent in this image? Choose either cat or dog.",
                image,
            ],
        }
    ],
    output_type=CFG('start: "cat" | "dog"'),
    max_tokens=12,
)

print(result.text)

Batch + Pydantic

from PIL import Image
from pydantic import BaseModel
from mlx_vlm_batch_outlines import batch_chat, load


class VisualSummary(BaseModel):
    primary_subject: list[str]
    subject_count: int
    setting: str
    short_description: str


model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")
cat = Image.open("cat.jpeg")
dog = Image.open("dog.jpeg")

results = batch_chat(
    model,
    processor,
    [
        [{"role": "user", "content": ["Describe this image.", cat]}],
        [{"role": "user", "content": ["Describe this image.", dog]}],
    ],
    output_type=VisualSummary,
    max_tokens=180,
)

for item in results:
    print(item.model_dump())

Video Chat

video_chat(...) is the thin direct video path. It samples frames across the whole video and runs one multimodal generation call.

from mlx_vlm_batch_outlines import load, video_chat

model, processor = load("mlx-community/Qwen2-VL-2B-Instruct-4bit")

result = video_chat(
    model,
    processor,
    video="path/to/video.mp4",
    prompt="Describe this video.",
    fps=1.0,
    max_pixels=(224, 224),
    max_tokens=100,
)

print(result.text)

Chunked Video Processing

video_chunk_process(...) treats a single video as many independent image-chat chunks.

For each chunk it:

1. slices the video by time
2. samples frames inside that chunk
3. converts those frames into a normal multi-image chat
4. runs chunk chats through batch_chat(...) in mini-batches

This is useful when you want independent chunk summaries instead of one global video answer.

from pydantic import BaseModel
from mlx_vlm_batch_outlines import load, video_chunk_process


class ChunkSummary(BaseModel):
    actions: list[str]
    scene: str


model, processor = load("mlx-community/Qwen3.5-4B-MLX-4bit")

results = video_chunk_process(
    model,
    processor,
    video="path/to/video.mp4",
    prompt="Describe what is happening in this chunk.",
    chunk_length_seconds=5.0,
    batch_size=4,
    fps=1.0,
    max_frames_per_chunk=8,
    output_type=ChunkSummary,
    max_tokens=120,
)

for item in results:
    print(item["chunk_index"], item["start_sec"], item["end_sec"], item["output"])

Each returned item looks like:

{
    "chunk_index": 0,
    "start_sec": 0.0,
    "end_sec": 5.0,
    "output": ...,
}

If output_type is structured, output is the parsed structured object. Otherwise it is the raw text string for that chunk.

Notes

• Default image resize is currently 224x224.
• Smaller image sizes can improve batching throughput significantly because image token count drops quickly with resolution.
• Homogeneous batches usually perform better than mixed multimodal shapes.
• batch_chat_stream(...) is not implemented.
• This package is Qwen-focused and not intended as a generic VLM abstraction layer.

Benchmark Notes

These are local measurements from the development machine, not a formal benchmark suite.

• 4B Qwen, homogeneous 3 x 1-image batch, 768x768:

  • sequential: about 19.25s
  • batch: about 15.63s
  • speedup: about 1.23x

• 4B Qwen, homogeneous 3 x 1-image batch, 384x384, short CFG output:

  • sequential: about 6.25s
  • batch: about 2.94s
  • speedup: about 2.13x

The practical takeaway is simple:

• batching helps more when image sizes are smaller
• batching helps more when the workload is homogeneous
• image resolution matters a lot because image token count grows quickly with width and height

Local Verification

There is a simple verifier script in:

• verify_mlx_vlm_batch_outlines.py

Run it with:

uv run python verify_mlx_vlm_batch_outlines.py