mlx-beam

B.E.A.M. — Batched Engine for Apple Metal. Light and modular inference engine, built on MLX.

Work in progress. See the feature table below and the changelog.

Install

uv tool install mlx-beam
beam serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-5bit --draft-model bundled

Inside a uv project: uv add mlx-beam, then uv run beam serve ....

beam doctor reports the Python, MLX, device and memory it sees (--json for scripts) and exits non-zero when MLX is missing or fails to load.

Serve

beam serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-5bit --port 8000 \
  --max-completion-tokens 4096 --min-response-tokens 512 --max-reasoning-tokens 8192 \
  --kv-bits 8

An OpenAI-compatible server (/v1/chat/completions, /v1/completions, /v1/responses, /v1/models) with Anthropic's Messages API beside it (/v1/messages; point ANTHROPIC_BASE_URL at it), /health reporting what was actually built and /metrics as plain-text counters. Flags follow mlx-lm's names where mlx-lm has one; beam serve --help lists them, and the configuration page has the same tables next to the request fields and what a checkpoint brings along. Off loopback the server wants --api-key. --draft-model bundled turns on speculative decoding with the draft head a checkpoint ships.

Features

Everything the engine does, with where each piece stands.

Extras are packages or optional installs beside the core; the core runs without any of them.

The engine reads standard MLX checkpoints. A checkpoint in the B.E.A.M. package layout (extras/manifest.json next to the shards; see the model cards under huggingface.co/p4ik) also tells it how its draft head was quantized and which KV prefill mode its profile was measured with. Which model families the engine handles with what - architecture, thinking, tool calls, vision, draft head - is on the models page.

Why it exists

Existing MLX servers either stop at the basics or grow things that have no place in an inference engine: a built-in game, a cloud path that arrives with an update. The ones we ran daily also had bugs where it matters most: prefix cache, batching under load, vision. B.E.A.M. keeps the core to the token path and fixes those paths at the source. Everything else is a package or an extra you choose to install; nothing ever ships in the core that you did not ask for.

Measured

Measured numbers are published as they are measured, with machine, model and date. Mac mini M4 Pro 64 GB, Qwen3.8-27B 5-bit with 8-bit KV, one request, 2026-09-26 (0.1.0a6): decode 11.4 tok/s at 12 000 generated tokens; prefill 115 tok/s over a 16 588-token prompt; a short request answered in 2.4 s while that prefill ran; 45 minutes under a steady stream, 5 057 requests, none failed. Speculative acceptance and batch numbers follow when they are measured the same way.

Contributing

See CONTRIBUTING.md. Rules for coding agents are in AGENTS.md.

License

Apache-2.0. Vendored components keep their own licenses; see NOTICE.