Kamiwaza-MLX 📦
A simple openai (chat.completions) compatible mlx server that:
- Supports both vision models (via flag or declared config detection) and text-only models
- Supports streaming boolean flag
- Has a --strip-thinking which will remove tag (in both streaming and not) - good for backwards compat
- Supports usage to the client in openai style
- Prints usage on the server side output
- Appears to deliver reasonably good performance across all paths (streaming/not, vision/not)
- Has a terminal client that works with the server, which also support syntax like
image:/path/to/image.png Describe this image in detail - Experimental direct multi-node bootstrap via
mlx.distributedwhenPAIRED_HOSTis provided; the current Kamiwaza TP=2 launcher/proxy flow is documented inMLX_TWO_NODE_README.md
Tested largely with Qwen2.5-VL and Qwen3 models.
Current MLX Stack
The Desktop candidate branch locks the MLX compatibility epoch as one
dependency unit. Dependency/import checks and offline regressions pass; semantic
vision and distributed runtime qualification remain open. These versions must move together because current mlx-vlm requires MLX
0.32 while mlx-lm remains on its 0.31 release line:
mlx==0.32.3mlx-lm==0.31.3mlx-metal==0.32.3mlx-vlm==0.7.4mlx-audio==0.5.7
Vision capability is fail-closed. If a model config declares a vision tower,
the server loads it through mlx-vlm or fails startup with the underlying
diagnostic; it never silently falls back to a language-only mlx-lm process.
Repository names alone do not declare vision capability. Use --text-only
for an intentionally text-only conversion that retains vision config metadata;
it is mutually exclusive with --vision. Requesting DFlash also remains an
explicit text-only mode and is logged as such for vision-capable checkpoints.
Earlier-epoch testing of mlx-community/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-4bit
used mlx_vlm generate for combined text, image, and audio inputs. This model
has not been requalified against the current lock. Native video is
not yet functional in this MLX path; the current upstream Nemotron MLX model
code raises NotImplementedError: Efficient video sampling is not implemented for Nemotron Omni yet.
Checkpoint code on the MLX runtime
On the MLX runtime of kamiwaza-mlx-server, --trust-remote-code is off
by default, including when this entry point runs on multiple ranks. The flag
allows checkpoint and processor-requested Python when loading the target and
any drafter. Their loaders may also request code from another repository.
Each loader resolves Hub IDs as it did before this change. With trust off,
the launcher checks the resulting directory before passing it to the loader.
Identical Hub IDs share a resolution when they use the same resolver. A
fallback from text to vision uses mlx-vlm's resolver and vision check. If
each resolver returns a config that selects the other loader, startup asks
for the local directory of the intended revision.
Run python -m kamiwaza_mlx.server from a trusted working directory, never
from inside a model directory. Python puts the working directory on the import
path by default for -m. The installed kamiwaza-mlx-server console script
uses its script directory for that import path entry.
| Path | Refused before load with trust off | Why | Measured (offline stand-ins, pinned packages) |
|---|---|---|---|
Text target (mlx-lm), including --text-only |
model_file in config.json |
mlx-lm imports the named file even with trust off. A config auto_map passes the directory check. A built-in tokenizer can still load when Transformers declines a custom AutoConfig. |
In 10 noninteractive llama attempts with the trust argument omitted, no auto_map code ran. The loader imported a declared model_file. |
| Vision target (mlx-vlm), including a fallback from text | model_file in config.json, a non-empty auto_map, audio_tokenizer_class or audio_tokenizer_name_or_path, or a model type listed below |
mlx-vlm can import the model file or delegate tokenizer and processor loading. The launcher checks before each call to the vision loader. | On qwen4_exp, auto_map code ran in 8 of 12 attempts with the trust argument omitted and 0 of 12 with trust_remote_code=False. Both settings imported a declared model_file. On qwen3_vl, tokenizer and config auto_map code ran with the weights step stubbed. |
Drafter (--draft-model or --dflash-draft-model) |
model_file in config.json |
Two drafter loaders import the named file. A drafter's auto_map alone does not trigger this check. The drafter check runs before the target loads. |
In 21 loads (seven per backend), dflash-mlx and mlx-vlm load_drafter imported model_file. Z-Lab imported no checkpoint module with Hub lookup redirected to the fixture. None of the three imported auto_map code. |
With trust off, all three paths check for model_file at any depth in the
config.json the loader opens, including spellings the filesystem treats
as equivalent. An unreadable config stops startup, and a drafter directory
must contain config.json. Vision also checks for auto_map and the two audio
tokenizer keys at any depth in each top-level JSON file. The vision check
applies its canonical file rules to its fixed list of loader filenames and
the config Transformers selects from configuration_files in config.json.
It refuses dangling links and requires each present canonical file to be
readable JSON of at most 64 MiB. It requires a JSON object except for
engram_token_map.json, which may have any JSON root type. The size limit,
JSON syntax check, and vision declaration scan still apply. For other top-level
JSON files the vision check skips unreadable or malformed files, warns for
oversized files, and refuses excessive nesting. A Python file that nothing
names can remain in any directory.
On the text path, a tokenizer auto_map also passes the directory check.
The launcher passes the explicit trust setting to mlx-lm's tokenizer and to
mlx-vlm's model and native drafter loaders. Opting in passes True and can
load a custom text tokenizer without prompting. Processor-only declarations
do not trigger the text directory check. With trust off, the Transformers guard refuses
remote-only classes and direct dynamic class imports, even if a dependency
asks for trust_remote_code=True. A loader that recovers using built-in code
can finish loading.
On refusal, the launcher exits 2 and attempts to write one error line. A
declaration refusal lists the file and matching fields, shortening long entries
and counting additional matches when the list is full. A Transformers
refusal names the model path or ID. With trust off, the launcher reports the
file and auto_map entry when it recognizes the declaration involved in a
text tokenizer failure.
Both kinds of refusal identify --trust-remote-code. An inspection error
names the path and the reason inspection failed.
The vision check refuses the following canonical types in mlx-vlm 0.7.4.
Their processors have paths that request or default to trusted tokenizer
loading. The check uses the top-level model_type when truthy, otherwise
speculators_model_type, then lowercases and applies mlx-vlm's remapping.
| Canonical type | Processor behavior |
|---|---|
got, inkling (including inkling_mm_model) |
Requests trust for its tokenizer. |
internvl_chat |
Requests trust when given a tokenizer name. |
minicpmo, minicpmv4_6 |
After a built-in tokenizer failure, takes the trusted fallback only when trust is enabled. |
moondream2 (including moondream1) |
Loads a tokenizer from another repository with trust enabled. |
moondream3 |
Can fall back to another repository's tokenizer. Preserves explicit trust and defaults to true when the argument is omitted. |
The original offline probe recorded 560 cases across MLX and oMLX, comprising 538 model or engine load attempts and 22 standalone tokenizer loads. The stand-in modules recorded import stacks and supplied minimal loader classes. The 21 drafter loads were measured separately.
Five of the 45 fetched catalog configs declare a non-empty auto_map, including
mlx-community/MiniMax-M2.7-4bit and mlx-community/DeepSeek-R1-0528-4bit.
Core lists both conversions as MLX variants. Their auto_map declarations
pass the default mlx-lm directory check.
None of the 45 fetched configs declares model_file.
Config fetches failed for three catalog repositories,
cognitivecomputations/Qwen3-235B-A22B-Thinking-2507-AWQ,
koushd/Qwen3-235B-A22B-Instruct-2507-AWQ and QuixiAI/Kimi-K2-AWQ.
These checks assume other processes running as the same user leave checkpoint
directories and files alone during startup. Some mlx-vlm processors render
checkpoint chat templates outside a sandbox. That behavior was identified by
source inspection and remains open under ENG-18520. The two-node launcher
(kamiwaza-mlx-distributed) and kamiwaza-mlx-distributed-dflash-server
retain their existing behavior. Neither distributed command was measured.
oMLX uses its own trust_remote_code model setting.
Choosing the runtime: MLX or oMLX
kamiwaza-mlx-server can serve a model itself (mlx-lm / mlx-vlm in this
process) or hand it to oMLX (Apache-2.0),
which runs from its own environment.
kamiwaza-mlx-server -m <model dir> # --runtime auto (default)
kamiwaza-mlx-server -m <model dir> --runtime mlx # this server, always
kamiwaza-mlx-server -m <model dir> --runtime omlx # oMLX, always
auto reads model_type from the model's config.json. The shipped
kamiwaza_mlx/config/runtimes.json gives the following choices.
model_type |
Model | auto picks |
|---|---|---|
deepseek_v41 |
DeepSeek V4.1, including Flash | oMLX |
qwen4_exp |
Qwen3.8-Flash-Next | oMLX |
A model that auto would otherwise serve with oMLX, launched with an option that selects MLX, such as --trust-remote-code, --ple-store, --draft-model, --dflash-draft-model or --distributed (see "Options of the MLX runtime on oMLX" for flags and environment variables) |
MLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice, for example runtime=mlx (auto: model_type=qwen4_exp, --ple-store given, mlx only) |
|
qwen4_exp whose config.json declares text_config.ple_storage or llm_config.ple_storage (the mlx-vlm external-table layout, our -extple directories) |
MLX: oMLX cannot load that layout. --runtime omlx on it is refused |
|
inkling2 |
Inkling 2 Small | oMLX (refused for now, see below) |
anything else, or no local config.json |
MLX |
The table is data. To change a default, edit that file or point
KAMIWAZA_MLX_RUNTIME_TABLE at your own copy. KAMIWAZA_MLX_RUNTIME sets the
option from the environment. The command line wins. The runtime line says
which runtime the launcher chose and why.
The launcher never falls back from one runtime to the other after a failure. If oMLX is selected and cannot be used, it exits with status 2 and prints the reason and the fix. When an option or checkpoint layout selects MLX, the runtime line names it before the model loads.
Upgrading from a build with fixed sampling defaults. On the MLX runtime,
each omitted or null sampling field now defaults to its valid checkpoint
value when available, even if the request sets the other fields. For example, a request
sending only temperature to Qwen3.8-Flash-Next now uses top_p=0.95 and
top_k=20 when those values are in the loaded checkpoint's
generation_config.json. To use the old defaults, start the server with
--sampling-defaults launcher or send
{"temperature": 1.0, "top_p": 1.0, "top_k": 0} in the request.
Earlier builds ignored the request's top_k. The launcher now applies it
on backends that support top-k. For a non-null top_k, the server rejects
negative integers, booleans and other non-integers. Requests with -1,
true or 20.0 return HTTP 422. Some servers accept -1 for no limit.
Send 0 for no top-k limit here.
Upgrading from a build without vision prompt reuse. On the MLX runtime,
a qwen4_exp vision-language model can now reuse matching prompt prefixes
by default when KV caching is enabled. The launcher checks reuse after the
model has loaded and any native vision drafter has passed its startup
self-test. Only MTP or no drafter qualifies. With --vlm-prompt-reuse on,
startup requires a working checkpoint manager. A checkpoint can still exceed
the memory cap. The default can keep up to two checkpoints whose cache buffers
total at most 8 GiB, and each active restore needs memory for its own copy.
Requests with images or media placeholder strings in the rendered prompt
read the full prompt. Use --vlm-prompt-reuse off for a full read on every
vision request.
On development commit 9174b27, the Qwen3.8-Flash-Next assistant messages
matched with reuse on and off in one Mac Studio run at temperature 0, both
without a drafter and with the MTP drafter. The comparison left out the
tool-call IDs the server generates per response. The measurements are under
"Qwen3.8-Flash-Next with a drafter".
Upgrading from 0.2.7 or earlier. --runtime is new in 0.2.9, the first
release published after 0.2.7 (0.2.8 was a version number used during
development and was not published). Up to 0.2.7 the launcher used only MLX.
With --runtime auto, a local checkpoint identified as DeepSeek V4.1 or
Qwen3.8-Flash-Next now goes to oMLX unless its layout or the launch options
select MLX. For a hub ID, auto starts with the table default (MLX in the
shipped table).
A host without the oMLX environment refuses an oMLX launch (exit 2,
with the install command) until the environment is installed or the launch
says --runtime mlx. A launch that passes other options of this server
(KV cache, tool parser, tracing and so on) for one of these models is refused
on oMLX too, naming the option; add --runtime mlx to keep it on this
server. Abbreviated options (--mod for --model, for example) are no longer
accepted by this server on either runtime; spell options out.
The new --draft-model option enables a native mlx-vlm drafter.
An explicit --draft-block-size must be at least 2 because one token comes
from the target and the rest are drafted. Startup requires a completed draft
round in up to three greedy text generations of up to eight tokens each.
The launch logs the round count on success and exits 2 if generation raises
or none of the three prompts runs a round.
On oMLX, a Qwen3.8-Flash-Next launch with a truthy mtp_enabled setting
requires an embedded draft-head tensor name in the checkpoint metadata.
On a launch that serves on the MLX runtime, kamiwaza-mlx-server reads its
bundled config/reasoning.json before it loads the model and exits 2 on a
file-read error, a JSON decoding error or a failed check of the table. A
launch that goes to oMLX does not read it.
Chat requests on the MLX runtime have these changes (see also "Reasoning controls").
- With a native drafter, an unset, null, zero or negative token limit uses
the served context window minus the prompt text token count, at least 1,
or
2**31 - 1if the window is unknown. Image tokens are excluded, so an image request can get more room than remains in the window. The vision path reportsstopfor an ordinary answer that reaches this bound. Generation keeps its previous limits for positive values or a launch without a native drafter. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens. - The
kamiwaza-mlx-serverchat endpoint on the MLX runtime now acceptsmax_completion_tokensas an alias formax_tokens. Version 0.2.7 ignored it. A null for either name now means unset, thoughmax_tokens: nullgave a422in 0.2.7. Two non-null values must match or the server returns a400. - 0.2.7 ignored
reasoning_effortandchat_template_kwargs. On a launch that passes them to the chat template, the server now passes the merged arguments through unless it has recorded reasoning facts for the model. A native vision-language launch adds its thinking-off default only when neither reasoning field is set. On Qwen3.8-Flash-Next,low,mediumandxhighsent alone turn reasoning on at that level.minimalis sent aslow,highasxhigh, andnoneturns reasoning off. On the Qwen native vision-language path,enable_thinking: truesent alone now reaches the template. - With stripping on and auto-detection on, whether an answer starts inside
a think block is now decided per request from the think tag that ends
its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next
with a drafter"). Visible effects: a request rendered with thinking off
(as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer
that could come back empty before; a vision-language stream without tools
rendered with thinking on no longer sends the reasoning and
</think>incontent(a block of more than 50 tokens with no think tag that a token limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the prompt, is withheld on every path, including one-piece answers and streams that used to send it; and a stripped vision-language stream that sees a</think>outside a think block now updates the process-wide detection result, as the other paths already did. Text and DFlash streams now keep visible text that comes before an opening tag in the same piece, and streams without tools recognize a closing tag split across pieces inside a block. Streams without tools withhold a block whose opening tag they recognized and that does not close; text and DFlash streams used to send a long one at the end. Vision-language streams match think tags in any case and remove a</think>that has no opening tag when it arrives in one piece. Think tags are now found in the original text, so a character whose lower case is longer (for exampleİ) no longer shifts the split; this applies with stripping or auto-detection off too. - Contradictory fields, reserved template argument names and unrecognized
Qwen3.8-Flash-Next efforts give a
400. Nestedenable_thinkingandreasoning_effortare allowed even when the renderer names them as parameters. When the fields are consistent and otherwise valid, the unrecognized-effort error lists the accepted names and mappings. When a comparison render attributes an error to the arguments, the400quotes the template error. In 0.2.7 a text renderer'sTypeErrorcould cause a retry withoutenable_thinking. Request validation can give a422before these checks. - A text renderer's
TypeError,ValueErroror Jinja template error that remains after tool fallback and field checks now gives a503quoting the error in place of an opaque500. A503from the native template renderer also quotes its error. - A launch that uses manual text formatting or mlx-vlm's non-native prompt
helper refuses reasoning fields with a
400. This includesenable_thinking, which 0.2.7 accepted there and ignored.
For an unforced minimax-m2 request on a text template that takes tools,
the server renders the same request with a unique name for each tool.
It uses the original prompt only if the second render succeeds, contains
every marker and matches the original exactly after restoring the real names.
In the renamed render each name must be the name of a JSON declaration
inside its own <tool>...</tool> block.
Otherwise it renders explicit instructions with the request's own arguments.
The parser reads only MiniMax's call format, so a template that presents
tools another way (Qwen3-Coder's, for example) gets explicit instructions.
A tools-only comparison that raises TypeError also uses this fallback.
MiniMax always uses explicit instructions for a forced choice, even on a
template that prints <tool> blocks.
Explicit instructions list every supplied tool in request order, even for a
forced choice. A missing declaration or missing instructions in the
rendered prompt give a 400. On these
explicit-instruction paths, a forced tool_choice naming a tool absent
from the request's tools is also a 400. With --tool-call-parser openai-json,
a forced choice now lists every supplied tool before the forced instruction.
Previously it showed only the selected function's schema. With tools enabled,
openai-json now returns a 400 for an otherwise valid tool with a function
member but a type other than function, with or without a forced choice.
In 0.2.7, openai-json could silently omit non-function tools.
From a 0.2.8 development build to 0.2.9. These changes are relative
to the last 0.2.8 development snapshot (fef2c29).
kamiwaza-mlx-distributedandkamiwaza-mlx-distributed-dflash-servernow exit 2 whenKAMIWAZA_MLX_RUNTIMEis set to something that is notauto,mlxoromlx. 0.2.8 ignored such a value there.- A single-node launch given
--distributed-world-size 1(or given--distributed-hostor--distributed-world-sizewhile the environment turns distributed serving off) has its port checked before the model is loaded, like any other single-node launch. In 0.2.8 a taken port was refused for such a launch only after the load. kamiwaza-mlx-server --helpprints the option list beside a settings file that cannot be read. 0.2.8 exited 2 without it.- If the null device cannot be used when the server ends, the non-blocking flag is left on its output pipe (see "Tying the server to its supervisor"). In 0.2.8 the server could hang at exit in that case.
- On a launch that serves on the MLX runtime,
kamiwaza-mlx-serverreads its bundledconfig/reasoning.jsonbefore it loads the model and exits 2 on a file-read error, a JSON decoding error or a failed check of the table. A launch that goes to oMLX does not read it. - With a native MLX drafter, omitting the limit no longer cuts the answer
to one token.
An unset, null, zero or negative limit uses the served context window
minus the prompt text token count, at least 1, or
2**31 - 1if the window is unknown. Image tokens are excluded, so an image request can get more room than remains in the window. The vision path still reportsstopfor an ordinary answer that reaches this bound. Generation keeps its previous limits for positive values or a launch without a native drafter. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens. - The
kamiwaza-mlx-serverchat endpoint on the MLX runtime now acceptsmax_completion_tokensas an alias formax_tokens. A null for either name now means unset, thoughmax_tokens: nullgave a422in 0.2.8. Two different non-null values now give a400where 0.2.8 ignoredmax_completion_tokens. - On
kamiwaza-mlx-server,reasoning_effortandchat_template_kwargsnow reach the model's chat template where the launch supports them. A native vision-language launch adds its thinking-off default only when neither reasoning field is set. On Qwen3.8-Flash-Next,low,mediumandxhighsent alone turn reasoning on at that level.minimalis sent aslow,highasxhigh, andnoneturns reasoning off. - With stripping on and auto-detection on, whether an answer starts inside
a think block is now decided per request from the think tag that ends
its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next
with a drafter"). Visible effects: a request rendered with thinking off
(as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer
that could come back empty before; a vision-language stream without tools
rendered with thinking on no longer sends the reasoning and
</think>incontent(a block of more than 50 tokens with no think tag that a token limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the prompt, is withheld on every path, including one-piece answers and streams that used to send it; and a stripped vision-language stream that sees a</think>outside a think block now updates the process-wide detection result, as the other paths already did. Text and DFlash streams now keep visible text that comes before an opening tag in the same piece, and streams without tools recognize a closing tag split across pieces inside a block. Streams without tools withhold a block whose opening tag they recognized and that does not close; text and DFlash streams used to send a long one at the end. Vision-language streams match think tags in any case and remove a</think>that has no opening tag when it arrives in one piece. Think tags are now found in the original text, so a character whose lower case is longer (for exampleİ) no longer shifts the split; this applies with stripping or auto-detection off too. - Contradictory
enable_thinkingandreasoning_effortvalues now give a400. - A reserved name in
chat_template_kwargson a template-backed launch now gives a400. Nestedenable_thinkingandreasoning_effortare allowed even when the renderer names them as parameters. - An otherwise valid Qwen3.8-Flash-Next request on a template-backed launch
with an unrecognized effort now gets a
400listing the accepted names and mappings. - A launch that cannot pass
enable_thinking,reasoning_effortorchat_template_kwargsto a template now gives a400if any is set. - A text renderer's
TypeErrorcould previously dropenable_thinkingon retry. When the request's own render fails and a comparison render without the request's arguments succeeds, the request now gets a400quoting the error. - A text renderer's
TypeError,ValueErroror Jinja template error that remains after tool fallback and field checks now gives a503quoting the error in place of an opaque500. A503from the native template renderer now quotes its error too. - For an unforced
minimax-m2request on a text template that takes tools, the server tries a second render with a unique name for each tool. It uses the original prompt only if that render succeeds, contains every marker and matches the original exactly after restoring the real names. In the renamed render each name must be the name of a JSON declaration inside its own<tool>...</tool>block. Otherwise it renders explicit instructions with the request's own arguments. The parser reads only MiniMax's call format, so a template that presents tools another way (Qwen3-Coder's, for example) gets explicit instructions. A tools-only comparison that raisesTypeErroralso uses this fallback. A forced MiniMax choice now lists every supplied tool in request order before the explicit instruction to call the selected function, even on a template that prints<tool>blocks. The server refuses instructions that omit a supplied tool or disappear from the rendered prompt. On explicit-instruction paths, a forcedtool_choicenaming a tool absent from the request'stoolsnow gets a400. The 0.2.8 build could answer with tools missing. With--tool-call-parser openai-json, a forced choice now lists every supplied tool before the forced instruction. Previously it showed only the selected function's schema. With tools enabled,openai-jsonnow returns a400for an otherwise valid tool with afunctionmember but a type other thanfunction, with or without a forced choice. - A
--draft-modellaunch now exits 2 if a greedy text self-test raises or all three test generations of up to eight tokens complete without a draft round. A successful self-test logs the number of completed rounds. - A
--draft-modellaunch now refuses--draft-block-size 1with exit 2 because mlx-vlm would draft nothing and end every answer after its first token. - With a truthy
mtp_enabledsetting, an oMLX launch of Qwen3.8-Flash-Next now exits 2 if the checkpoint metadata lists no embedded draft-head tensor. The 0.2.8 launcher did not check the metadata.
The oMLX environment
oMLX is not a dependency of this package. It requires an exact mlx build
that differs from this package's, and it is published as release wheels on
GitHub, not on PyPI. Install it once per host, beside this environment:
python -m kamiwaza_mlx.omlx_backend install --prefix <root>/omlx-venv # needs uv
python -m kamiwaza_mlx.omlx_backend check
install downloads the pinned oMLX release wheel, verifies its SHA-256, and
then installs it with uv together with the locked dependency set shipped in
this package: 108 packages from PyPI at exact versions and four built
from pinned git commits. Only the oMLX wheel is checked against a hash; the
other packages are pinned by version or commit, not by hash. It needs network
access and refuses to touch an existing environment. An installation that
fails part-way (a network error while the dependencies are fetched, for
example) can be finished by running the same install command again: the
installer marks the directory when it begins and when it has finished, and a
directory it began and did not finish is installed into again. Until then
check and every launch refuse that environment as incomplete, by name.
Nothing is deleted by the installer; a directory it did not begin is left as
it is. The pin (version, wheel URL, hash,
lock file) is in kamiwaza_mlx/config/omlx.json; today it is oMLX 0.7.3
with mlx==0.32.4 on Python 3.12. The launcher looks for the environment in
KAMIWAZA_OMLX_HOME, or omlx-venv next to the environment this package is
installed in, and refuses any other oMLX version.
Inkling 2 Small (inkling2) needs oMLX support that is not in any
published release yet (upstream commit adc3daa6, PR #5229). Until the pin
moves to a release that has it, the launcher exits and says so. The regular
MLX runtime cannot serve it either: mlx-lm and mlx-vlm at this package's pins
have no inkling2 model type.
What a launch on oMLX does
- Everything oMLX reads or writes is under one state directory per model:
KAMIWAZA_MLX_OMLX_STATE_DIR/<model directory name>-<key>(default~/.kamiwaza-mlx/omlx/...), where the key is derived from the model directory's full path, so two models whose directories share a name do not share state, and the same model finds its cache again. One server at a time uses a state directory: a second launch of the same model directory while the first is running exits 2; give it its ownKAMIWAZA_MLX_OMLX_STATE_DIRto run two copies. It holds the oMLX profile, a view of the model made of links (no weights are copied), and the prompt-prefix cache. Nothing is written to~/.omlx, andOMLX_*environment variables are not passed on. - oMLX's start-up clean-up of distributed rank processes is switched off.
Left on, it terminates ranks recorded under
~/.omlxwhose coordinator has exited, and those can belong to another oMLX installation on the same Mac. - oMLX's own
/healthanswers without a key and includes the model's name and memory figures; the MLX runtime's/healthreturns the status only. - Only inference routes are served (
/health,/v1/models,/v1/chat/completions,/v1/completions,/v1/embeddings,/v1/messages,/v1/messages/count_tokens,/v1/responsesand/v1/responses/<id>,/v1/tokenize,/v1/detokenize). oMLX's administration pages and API answer 404, and its WebSocket routes (realtime audio) are closed before they are accepted. - For DeepSeek V4.1 a request is held to the model's native context: prompt
tokens plus the output budget plus a small reserve may not exceed
1,048,576, or the request is answered 400 before anything is generated.
The answer is oMLX's own overflow error (code
context_length_exceeded, message opening "Prompt is too long: N tokens > 1048576 maximum context length"), so clients that compact a conversation on overflow do so; it names the field the route takes its budget from (on chat both spellings,max_tokensandmax_completion_tokens, since oMLX's parsed request does not record which one was sent). The check uses what oMLX itself resolved: the request oMLX parsed and validated and the budget field its route generates with, at the point where each generating route (/v1/completions,/v1/chat/completions,/v1/messages,/v1/responses, streaming or not) checks the prompt against the context window. The launcher does not parse request bodies. At start-up it verifies that the installed oMLX still has that point and exits 3 naming what differs if it does not. - No request body over
KAMIWAZA_MLX_MAX_REQUEST_BODY_MB(default 32 MiB) reaches oMLX, on any model, whatever its content type and whether it has aContent-Lengthor is chunked: it is answered 413. The bytes are counted as they arrive and never buffered by the launcher. This runs before the key check, because the web framework reads a body before it checks the key. - Browser clients (pages that call the server from JavaScript): on oMLX the
browser origins
http://127.0.0.1:<port>andhttp://localhost:<port>are allowed and no other, whatever--hostis; there is no setting for it. The MLX runtime sends no cross-origin headers at all, so a browser blocks every cross-origin read from it. A page served from another origin therefore works against neither runtime directly. Non-browser clients are not affected. - Requests need no API key unless one is configured (see "API key" below).
- The model is loaded at start-up.
/healthanswers 503 ("status": "loading") until it is loaded and 200 afterwards. If the load fails, the process prints the reason and exits with status 4; it never stays up without a model. While the model is loading, an inference request gets503at once, withRetry-After: 5and{"error": {"code": "model_loading", ...}}; it is not held until the load ends. This answer, like the404for routes that are not served, comes before any API-key check. Both are given by a plain ASGI wrapper that looks at the method and the path only and does not wrap a request it lets through (the body-size count above is the one thing the launcher puts around a request), so a streamed answer and a client that disconnects are oMLX's own to handle. A stop signal (SIGTERM, SIGINT) during the load ends the process within a second or two (status 143 or 130) without finishing the load. - The model is served under its directory name, one request at a time; later requests wait.
- The shipped oMLX profile turns reasoning off for Qwen3.8-Flash-Next.
Send
"enable_thinking": true(or the same key inchat_template_kwargs) to turn it on. On the default MLX launch, a recognized effort level or alias alone also turns it on. DeepSeek V4.1 reasons unless the request turns it off, as in the deployment it was validated with. See "Reasoning controls" for the levels and the differences between runtimes. - Drafting on oMLX uses a multi-token-prediction head stored inside the
checkpoint, turned on with
KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. For Qwen3.8-Flash-Next, a truthymtp_enabledsetting requires a tensor name starting withmtp.,language_model.mtp.,model.mtp.ormodel.language_model.mtp.. The launcher checks the safetensors index if it can read it as a JSON object, or the shard headers otherwise. If it finds no matching name, it exits 2. To serve without drafting, setmtp_enabledtofalse. To use a separate drafter directory, see "Qwen3.8-Flash-Next with a drafter". /v1/modelsreports the context oMLX serves, which is the model's own (262,144 for Qwen3.8-Flash-Next). The MLX runtime reports 131,072 for the same directory because that is the window it serves by default (--kv-cache-max-tokens, at most 128K unless raised). Each number is true for its runtime.- The launcher process becomes the oMLX process (same PID).
| Variable | Meaning |
|---|---|
KAMIWAZA_OMLX_HOME |
The oMLX environment |
KAMIWAZA_MLX_OMLX_STATE_DIR |
Parent of the per-model state directories |
KAMIWAZA_MLX_OMLX_SSD_CACHE_MAX |
Size of the on-disk prefix cache (default 20GB) |
KAMIWAZA_MLX_OMLX_MEMORY_CEILING_GB |
oMLX memory-guard ceiling (DeepSeek V4.1 defaults to 400, for a 512 GB Mac) |
KAMIWAZA_MLX_MAX_REQUEST_BODY_MB |
The largest request body passed on to oMLX, in MiB (default 32; 1 to 512). A larger body is answered 413. Text requests are far below the default; a vision model that is sent images or video as base64 may need more. 512 is oMLX's own limit, which the launcher does not change; a larger value is refused at start (exit 2) |
KAMIWAZA_MLX_OMLX_MODEL_SETTINGS |
JSON object merged into the oMLX model settings, e.g. {"mtp_enabled": true}. is_pinned cannot be turned off. A truthy mtp_enabled on Qwen3.8-Flash-Next requires a draft-head tensor name in the checkpoint metadata (exit 2 if absent) |
API key
KAMIWAZA_MLX_API_KEY makes the server require a key, on either runtime:
every route except GET /health then needs Authorization: Bearer <key> (or
x-api-key: <key>), and answers 401 without it. CORS preflight (OPTIONS)
requests carry no key and are let through. With no key set, the
server is open, as before. The earlier name KAMIWAZA_MLX_OMLX_API_KEY is
still accepted; if both are set to different values the launcher exits 2.
The key can also be written in a .env file in the directory the server is
started from; the process environment wins over the file. That file is read
the same way on both runtimes and by the launchers that refuse a key. oMLX's
own OMLX_API_KEY is not used: every entry point, on either runtime, exits 2
while it is set (in the environment, in the settings file, or passed with
--env) and names the variable to use. The rule behind it: no entry point
starts listening without authentication while a key variable it knows about
is set. The start-up output says whether a key
is in force: on the MLX runtime API key authentication: enabled (every route except /health). or API key authentication: off.; on oMLX kamiwaza-mlx: inference API key: required or kamiwaza-mlx: inference API key: off (requests are not authenticated). (oMLX's own API key authentication: enabled line, which it would print on every launch because the launcher always sets its internal management key, is reworded to oMLX management key: set (internal; ...).)
A program that imports kamiwaza_mlx.server to serve inside its own process
is held to the same rule, and it fails closed: the key is decided from the
process environment and the settings file together, so a key in either is
enforced, and where the configuration cannot be trusted (a settings file
that cannot be read, OMLX_API_KEY set, two key variables that disagree,
a key in a ./.env that a named file replaces) the import ends with
SystemExit(2) and no server starts. Such a program cannot turn the key
off by leaving the settings file unloaded.
The key is never logged and is not written to the launch description; on
oMLX it is in the profile's settings.json (mode 600). The key applies to
kamiwaza-mlx-server. The two-node launcher (kamiwaza-mlx-distributed) and
kamiwaza-mlx-distributed-dflash-server serve through mlx_lm.server, which
checks no key, so they refuse to start while the variable is set.
GET /health exists on both runtimes. On oMLX it answers 503 while the
model is loading and 200 once it is loaded. The MLX runtime does not listen
until its load step has returned, so there 200 means the server is up; a
large model loaded lazily still reads its weights on the first request, which
is slow.
Tying the server to its supervisor
By default the server keeps running if the process that started it dies. A supervisor that must never leave a model server behind passes one of:
| Option | The server stops when |
|---|---|
--exit-with-parent |
the process that started the launcher exits |
--parent-pid <pid> |
the process with that PID exits (for a supervisor that is not the direct parent, for example behind a wrapper or nohup) |
The contract, on both runtimes:
-
The watched PID is fixed when the launcher starts and is never re-read. The launcher prints
kamiwaza-mlx: this server stops when process <pid> exits. -
If that process does not exist at start, or
--exit-with-parentis used when the parent is already PID 1, the launcher exits 2 and starts nothing. -
When the watched process exits, however it exits, the server prints
kamiwaza-mlx: the supervising process <pid> is gone; stopping this server.(on oMLX always; on the MLX runtime while the web server runs, and before that the process ends without the line), sends itself SIGTERM, and if it is still running 5 seconds later exits with status 5. Those 5 seconds are fixed:KAMIWAZA_MLX_STOP_GRACE_SECONDSdoes not lengthen them, so with a supervisor gone a request in flight gets at most 5 seconds. During a model load it stops at once. -
The watch covers the model load and, on oMLX, lives in the process that remains after the launcher replaces itself (same PID as the launcher).
-
It watches for the process exiting, not for a signal sent to it: a supervisor that is stopped (SIGSTOP) or hung still counts as alive.
-
Output is best effort when the pipe it goes to is full, and no write waits on it for longer than a set bound (below). A supervisor usually reads the server's stdout and stderr through pipes; once it is gone a write to them fails, and if it stops reading, a blocking write waits forever once the pipe is full. So, when (and only when) descriptor 1 or 2 is a pipe or a socket, two processes make that descriptor non-blocking and replace
sys.stdout/sys.stderrwith a thin wrapper around it:Process Best-effort output kamiwaza-mlx-serveron the MLX runtime (console command orpython -m kamiwaza_mlx.server)yes The oMLX process the launcher becomes ( --runtime omlx)yes kamiwaza-mlx-distributed(it starts the ranks and the proxy on the same pipe and is replaced bymlx.launch)no: ordinary blocking output kamiwaza-mlx-distributed-dflash-server(one rank among others on the same pipe)no: ordinary blocking output A program that imports kamiwaza_mlx.serverno: the import replaces no stream and installs no error hook and no signal handler, and its refusals are not hard exits (see "Importing the server module" below for what it does do) In those two processes a write goes straight to the descriptor. If the pipe is full the write waits for it for at most 50 ms in total, then drops the rest. For the next second no write waits: each puts what fits into the pipe and drops the rest. So once a write has waited its full 50 ms in vain, no thread waits again for a second. That does not bound a reader that is slow but gets there: a write that the reader takes after, say, 30 ms waits those 30 ms and loses nothing, and so does the next one, so such a reader slows the server down by that much per write. No write raises, and
flushdoes nothing. Once that second has passed, a write adds one line saying what was lost:[kamiwaza-mlx: output could not be delivered on <stderr>: N writes (M bytes) were dropped], at most once every 5 seconds; loss right before the process exits is not reported. A line can be cut short where the pipe filled up, and so can that line. The bound for a reader is those 50 ms per write: output is complete when the reader takes every write within 50 ms (tested with 5,000 log lines; an 8 MiB burst of 1 KiB lines printed as fast as possible to a reader that reads as fast as it can was delivered whole when measured on the development machine, macOS on Apple silicon; the figure depends on the machine's pipe size and scheduling). A reader that pauses longer or starts late loses what did not fit in the pipe meanwhile, and a single write too large to be taken within 50 ms loses its tail. A terminal and a regular file are left exactly as they are, descriptor and stream. -
Side effects of the non-blocking flag, which belongs to the pipe end and not to the process. While the server runs, anything else that writes through the same pipe end sees it: native code in the server process gets "would block" where it used to wait (intended), and so does a child process or a sibling in the same shell pipeline that shares the server's stdout or stderr. The reading end is not affected. Give the server its own pipe if something else must write to it with blocking semantics while the server runs.
-
The flag is set late and taken back.
kamiwaza-mlx-serversets it only once the runtime choice has decided that this process serves on the MLX runtime, and the oMLX process sets it when it starts: a launch refused by the runtime choice (a bad option, a setting, a key, a taken port, a missing oMLX environment) leaves the pipe as it was. The flag is cleared again, where this process set it, when the process ends by a normal exit, a refusal or any other exit with a status of its own (load failed, stop deadline, supervisor gone, start-up error), and by SIGTERM after the web server's own stop. Just before, the process points its descriptors 1 and 2 at the null device, so from that moment nothing it writes reaches the pipe (a write to a blocking pipe that is full and unread would wait for good, and the interpreter still writes while it shuts down). The null device is opened when the process replaces its output streams, not when the process ends. If descriptor 1 or 2 cannot be pointed at it (the null device could not be opened, or the process has no file descriptor left), no flag is cleared: a process that cannot hang matters more than the flag. It is also not cleared where the process gets no chance to: after SIGKILL or another signal that ends a process by default (SIGHUP, SIGQUIT and the like), after a crash in native code or code that callsos._exiton its own, and after a SIGTERM during the MLX runtime's start-up (from the runtime choice until the server serves, the model load included), which ends the process at once, by design. A program that writes to the same pipe after one of those finds it non-blocking; a supervisor that runs further commands on a server's pipe can clear the flag itself (os.set_blocking(fd, True)). -
Several servers on one pipe. Each sets the flag if it is not set and clears it on leaving only if it set it; every write first sets it again if another server has cleared it meanwhile. So after one of them leaves, the pipe is blocking until the next write of one that still runs, and it is blocking after the last has left, in whatever order they leave. One window remains: a server that leaves at the very moment another is between that check and its write (the check is made before a write and again after a wait), with the pipe full and unread, lets that one write wait for the reader.
-
Importing the server module. For a program that imports
kamiwaza_mlx.server, the module body runs startup andmain_entry()serves. Argparse handles usage errors. Startup refusals, including native drafter load and self-test failures, raise a catchableSystemExit. Other target-load errors and interrupts propagate. Once installed with trust off, the Transformers guard stays active in that process. The settings file (the one named with--distributed-env-file, else./.env) is loaded where it always was (when distributed settings are resolved, after the options have been parsed), into the environment for names not already set; it is not loaded early for every setting as the console command does, so an option default taken from the environment does not see it. Differences from before: the file is read by the parser described under "Every setting on every path" (a line that is not a setting is an error instead of being skipped; only matching quotes are removed; a leadingexportand a byte-order mark are accepted); the refusals for a named file that is missing or empty and for the option given twice apply, at the point where the key is decided, which for an importer is after the model has loaded; and the key is decided as on the console command, from the environment and the settings file together (the file is read in place for this, not loaded), soKAMIWAZA_MLX_API_KEYin either is enforced, an empty variable in the environment does not cancel a key in the file, and the refusals forOMLX_API_KEYand for a key in a./.envthat a named file replaces apply. With--exit-with-parentor--parent-pidon its command line the importing process is watched and ended like the console command's.main_entry()itself reads no settings file. -
On the two distributed commands a full, unread pipe can still block ordinary output (logging, the ranks' own lines). Their refusals and start-up failures are armed from another thread before their line is written, so those exits do not depend on the pipe.
-
Without either option nothing is watched; running the server from a terminal is unaffected.
Exit status
| Status | Meaning |
|---|---|
| 1 | An error at start-up that no refusal covers: one line start-up failed: ..., then the traceback. On kamiwaza-mlx-distributed also a required setting that is missing or a local preparation step that reported its failure (one line; a step that failed with an error nobody handled prints the traceback too). On oMLX also a port taken between the launcher's check and oMLX's own bind: oMLX's error, before the model is loaded |
| 2 | Refused: bad option or option value, unreadable config.json, unusable runtime, missing, incomplete or mismatched oMLX environment, unsupported model or layout, conflicting keys, a bad setting, a settings file that cannot be read, a port that cannot be bound (for a wildcard host such as 0.0.0.0, also a port another process holds on the loopback address alone; checked before the load for every launch on oMLX and every single-node launch on the MLX runtime, --distributed-world-size 1 included, and not for a distributed one, where a rank that does not serve is given the same port), supervisor already gone. Nothing is served. On the MLX runtime with trust off, the launcher checks the drafter before the target loads. It checks each target path before its loader runs, including a vision fallback after a text attempt. Transformers can refuse during loading. A drafter that cannot be resolved, inspected or loaded also exits 2. Requesting --ple-store on a text route exits 2 before weight loading. A table supplied through --ple-store that mlx-vlm rejects as incompatible exits 2. If checkpoint inspection refuses the checkpoint's own PLE manifest, startup exits 2; if inspection passes and mlx-vlm rejects that manifest during loading, it is a model-load error and exits 4. A speculative drafter that loads is also refused if a test generation raises or none of three test generations completes a draft round (see "Native speculative decoding"). Drafter initialization and test-generation failures release the loaded target model. The eligibility and manager checks for --vlm-prompt-reuse on can also refuse the launch after the model and any vision drafter have loaded and the drafter's self-test has passed. On the MLX runtime, a port taken between the check and the web server's own bind is the same refusal, after the load (on oMLX see status 1) |
| 3 | oMLX start-up could not be made safe: a start-up guard could not be applied, the profile could not be written, or the supervising process exited in the moment between the launcher's check and oMLX starting |
| 4 | The model could not be loaded, including an ordinary vision-model load failure. MLX summarizes the error and up to four of its causes on one line and logs the traceback at debug level. oMLX reports when it finds no pinned model to load. If loading a pinned model raises an exception, it reports the exception message or, if empty, its type. |
| 5 | The supervising process exited and the server did not stop within 5 seconds of its own SIGTERM |
| 143 / 130, or ended by the signal | Stopped by SIGTERM / SIGINT. See the next table: the same stop reaches a supervisor in more than one form |
What a supervisor sees after a stop
The rule: a stop that the web server completes by itself ends the way the web server ends it, and a stop that the launcher has to complete ends with an exit status. Concretely:
- While serving, the first SIGTERM or SIGINT starts the stop deadline,
KAMIWAZA_MLX_STOP_GRACE_SECONDS(default 10; 0 to 3600), on either runtime. If every request finishes in that time the web server stops by itself; if not, the process exits when the time is up with 128 plus the signal (143, 130), also when its output can no longer be written.KAMIWAZA_MLX_OMLX_STOP_GRACE_SECONDSis accepted as another name for the setting; the two set to different values are refused. - During a model load a stop is immediate.
- When the supervising process is gone (
--exit-with-parent,--parent-pid), the server sends itself SIGTERM, which is a stop like any other and so starts the stop deadline, and it exits with status 5 if it is still there 5 seconds later. Whichever comes first ends it: the web server's own stop, the stop deadline (when the grace is under 5 seconds), or status 5.
A supervisor that reads the wait status (Python's Popen.returncode, for
example) therefore sees an orderly stop and a stop by the deadline
differently, where a shell shows 143 for both. Treat "killed by SIGTERM"
and exit status 143 as the same outcome: the server stopped because it was
asked to. The cases that have been observed:
| Case | Wait status | Observed |
|---|---|---|
| SIGTERM while serving, orderly stop, either runtime | killed by SIGTERM (returncode -15) |
live, both runtimes |
| SIGTERM while serving, a request still in flight when the grace ends | exit status 143 | live on oMLX; on the MLX runtime by test with a real web server |
| SIGINT while serving on oMLX, orderly or by the deadline | exit status 130 | live |
| SIGINT while serving on the MLX runtime, orderly stop | exit status 0 | live |
| SIGINT while serving on the MLX runtime, a request still in flight when the grace ends | exit status 130 | by test with a real web server |
| SIGTERM / SIGINT while oMLX is loading the model | exit status 143 / 130 | live |
| The model could not be loaded | exit status 4 | live |
| Refused at start | exit status 2 (3 for an oMLX start-up guard) | live |
| The supervising process was killed | gone within 0.1 to 6.1 seconds, both runtimes | live; the status has no observer left |
Not observed, from the code: SIGTERM while the MLX runtime is loading the model ends the process killed by SIGTERM; SIGINT at any point before the MLX runtime serves (during the load, or between the load and serving) ends it with exit status 130.
Every setting on every path
docs/settings-parity.md lists the settings the package reads (a test finds them by scanning the source for environment variables read through its helpers and for the options of its parsers, and fails when one is in no row). It sets them (the API
key, distributed settings, listener, runtime choice, the MLX-only options,
oMLX settings, oMLX's own OMLX_* variables and the rest) against every way
it can arrive (command line, process environment, ./.env, --env passed to
ranks, the selection table or an oMLX profile override) for each of the four
entry points. Every cell is honored, refused by name, or not applicable with
the reason; a variable that does not apply to an entry point is named in a
start-up line (set but not used by ...). The table is generated from
kamiwaza_mlx/settings_parity.py, and a test fails when a setting is added
without a decision.
./.env in the directory the server is started from is loaded into the
environment before any setting is read, on both runtimes; what is already in
the process environment wins (an empty KAMIWAZA_MLX_API_KEY in the
environment does not cancel a key in the file). A file named with
--distributed-env-file is read in place of ./.env, as before; nothing is
merged from the two, and a start-up line says which file is in force when
both exist.
A settings file is read strictly, by one parser for all four entry points
and the distributed runtime. A line is NAME=value, with optional
surrounding whitespace, an optional leading export, and optional matching
quotes around the value; blank lines and lines that start with # are
skipped. Nothing after the = is treated as a comment: in
KAMIWAZA_MLX_API_KEY=secret # prod the key is secret # prod. Because
that is legal and seldom meant, a value that contains # is pointed out at
start-up by file, line and variable name (never by value). A value that
begins with # (NAME= # comment) would be nothing but the comment: for a
key variable that is refused, naming file, line and variable; for any other
it is pointed out the same way.
Where the launcher cannot be sure what was configured, it exits 2 naming the
file and does not start: a settings file that exists and cannot be read or
decoded; a line that is not a setting (the message gives the line number and
never the line); a file named with --distributed-env-file that does not
exist or holds no settings, also when it names ./.env (it does not fall
back to ./.env); --distributed-env-file given more than once; and a
./.env that is not read because another file was named, while it holds an
API key that the named file and the environment do not.
kamiwaza-mlx-server --help (or -h) starts nothing, so it is not refused
for the settings file (./.env, or the one named with
--distributed-env-file): it prints the option list, preceded by a line
that says a launch with these settings would be refused and why.
kamiwaza-mlx-distributed and the DFlash distributed server do not
load ./.env; they refuse a key or KAMIWAZA_MLX_RUNTIME=omlx written
there, and name any other setting found only there in a start-up line.
They also refuse a KAMIWAZA_MLX_RUNTIME that is not auto, mlx or
omlx (a typo), in the environment or in ./.env, as kamiwaza-mlx-server
does: both always serve with MLX, but a misspelled value means the
configuration is not what its author meant. kamiwaza-mlx-distributed
refuses it in an --env assignment too. (Up to 0.2.7 the MLX server read that
file only for its distributed settings and, as a side effect, for anything
it happened to read later.)
Options of the MLX runtime on oMLX
No option is dropped. Each one is either already what oMLX does, or makes
auto choose the MLX runtime, or stops the launch naming the option and the
runtime.
| Option | On oMLX |
|---|---|
-m, --host, --port, --exit-with-parent, --parent-pid |
Used |
--vision |
Accepted: oMLX detects a vision checkpoint itself |
--strip-thinking |
Accepted: oMLX returns reasoning in a separate field |
--trust-remote-code |
With the shipped table, the flag selects MLX when auto would otherwise choose oMLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice. Explicit --runtime omlx refuses the flag. To allow checkpoint code on oMLX, remove the flag and set KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"trust_remote_code": true}'. |
--ple-store |
With --runtime auto, selects the MLX runtime (see the table above). With an explicit --runtime omlx: refused. oMLX keeps the n-gram table of Qwen3.8-Flash-Next on disk by itself (the launcher sets qwen4_ple_ssd_offload), but always the table in the model's own checkpoint. Pairing weights with the table of another conversion needs --runtime mlx |
--draft-model, --draft-kind, --draft-block-size, --draft-queue-timeout, --draft-stream-send-timeout, --dflash-draft-model |
With --runtime auto, select the MLX runtime. With an explicit --runtime omlx: refused. oMLX drafts with the head embedded in the checkpoint: on by default for DeepSeek V4.1, and for other models through KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. A separate drafter directory needs the MLX runtime |
--vlm-prompt-reuse, --vlm-prompt-reuse-checkpoints, --vlm-prompt-reuse-max-gb |
With the packaged runtime table, these settings do not select a runtime. If auto chooses oMLX or you select oMLX explicitly, supplying any of these flags or a nonblank value for its environment variable stops the launch, even for --vlm-prompt-reuse off. oMLX manages its own prompt cache. Use --runtime mlx to apply these options. There, --vlm-prompt-reuse on also requires a model type on the checked list. |
--sampling-defaults |
oMLX manages its own sampling defaults. With the packaged runtime table, this option does not influence runtime selection. If auto or an explicit choice selects oMLX, supplying the flag or a nonblank value for its environment variable stops the launch, even for model. |
--text-only |
Refused |
--no-distributed |
Accepted: oMLX serves on one node |
a word that is neither an option nor an option's value (... 8080) |
Refused, naming it |
--distributed and the other --distributed-* options |
Under --runtime auto they select the MLX runtime, so the launch does not reach oMLX. With --runtime omlx they are refused by name |
distributed MLX turned on through the environment or a .env file in the working directory (KAMI_DIST_ENABLED, MLX_DISTRIBUTED_ENABLED, PAIRED_HOST, MLX_DISTRIBUTED_HOST, MLX_DIST_HOST, or PAIRED_WORLD_SIZE / MLX_DISTRIBUTED_WORLD_SIZE above 1) |
Under --runtime auto it selects the MLX runtime, named in the runtime line. With --runtime omlx it is refused, naming the variable: oMLX would start a separate single-node server on every rank. A distributed variable that does not turn the mode on (RANK, PAIRED_PORT, MLX_DISTRIBUTED_TIMEOUT and the like) has no effect on oMLX and is named in the set but not used by ... line |
| every other option (KV cache, tool parser, tracing, ...) | Refused: it configures this server, not oMLX |
Sampling defaults
For stock text and vision-language generation on the MLX runtime, each of
temperature, top_p and top_k is chosen independently. A non-null
request value wins. For an omitted or null field, the server uses the
checkpoint's value, then falls back to temperature=1.0, top_p=1.0 or
top_k=0 (no top-k limit). An explicit zero counts as a value, as does
1.0 for temperature or top_p. In a request trace, request normally
holds the validated request with schema defaults before sampling defaults
are applied. Omitted sampling fields are null. If the server cannot
serialize the request, this field holds only model.
resolved_sampling holds the resolved values of temperature,
top_p and top_k.
| Flag | Environment variable | Default | Meaning |
|---|---|---|---|
--sampling-defaults {model,launcher} |
KAMIWAZA_MLX_SAMPLING_DEFAULTS |
model |
Use the checkpoint's sampling defaults. Select launcher to use the fixed defaults for omitted or null fields. |
In model mode the launcher reads generation_config.json from the loaded
vision model's model_path if it names an existing directory. Otherwise it
uses the launcher's resolved model directory. Its last choice is the loaded
tokenizer's name_or_path if that names an existing directory.
If no directory is known, the startup log reports model directory unknown
and the launcher uses its fixed defaults.
For checkpoint values, temperature must be finite and nonnegative,
top_p must be greater than zero and at most one, and top_k must be a
nonnegative integer. An invalid field produces a warning and uses its launcher
default. The startup log lists each default and its source. The launcher
ignores do_sample and other keys when choosing these defaults. Both pinned
model loaders also open the file. An unreadable file can prevent startup, and
mlx-lm can also reject JSON that is not an object.
For request values, a supplied, non-null top_k must be a nonnegative
integer. The request accepts numeric temperature and top_p outside the
checkpoint ranges. The pinned mlx-lm and autoregressive mlx-vlm samplers
apply top-p filtering only for 0 < top_p < 1. Zero temperature selects
greedy decoding. With a negative temperature the sampler divides the scores
by that value, reversing their order. If a top-k or top-p filter is active,
including one taken from the checkpoint, the tokens it filtered out then
score highest. Autoregressive mlx-vlm, including Flash-Next, rounds
positive temperatures below 0.01 up to 0.01.
For both requests and checkpoints, top_k=0 disables the cutoff. The
launcher collects vocabulary sizes from model configuration, output weights
and tokenizer length. A cutoff at or above the largest known size also means
no limit, with a startup warning for a checkpoint value. With no known size,
the launcher passes the cutoff through. When an autoregressive sampler applies
top-k filtering, a positive cutoff at or above the actual output width raises
an error. With an EAGLE-3 drafter whose vocabulary is smaller than the
target's, mlx-vlm applies the same sampler to the draft logits, so an active
cutoff at or above the draft vocabulary size raises the same error. At zero
temperature the sampler returns the top token without applying the filter.
The dflash-mlx text backend discards both the temperature and the sampler,
so none of temperature, top_p or top_k applies there, whether supplied
by the request or the checkpoint. The --dflash-backend auto setting can
also select this backend when installed.
DiffusionGemma in the pinned mlx-vlm ignores top_p and top_k from both
requests and checkpoints.
Tool-call parsing
With tools enabled on MLX, qwen-xml first applies its existing whole-answer
grammar to the visible text. An answer consisting of valid calls and whitespace
returns those calls in order with message.content set to null.
In mixed text, the parser takes every complete <tool_call>…</tool_call> block
outside reasoning, including blocks in prose or Markdown fences. It skips paired
<think> spans left in the visible text and everything from an unmatched opener
to the end. It skips the whole mixed answer as unfinished reasoning if the
generation prompt ends in <think> after trimming trailing whitespace and the
raw output has no </think>. Both tag checks ignore ASCII case.
Each block uses the call-only grammar to check its function name and arguments.
If a block fails to parse, or any individual gap outside reasoning
contains a wrapper tag matching </?tool_call(?![A-Za-z0-9_-]), the whole set
is discarded. With no calls, the server keeps the visible text unchanged and
runs its existing validation on it. Tool choice and parallel-call limits apply
to extracted calls. A tool-output error produces HTTP 502 in JSON or an
invalid_model_tool_output event followed by [DONE] in an HTTP 200 stream.
Read both message.tool_calls and message.content. Removing the blocks leaves
the original text and retained reasoning in order, with outer whitespace trimmed
and an empty result represented by null. A buffered stream sends reasoning,
then any remaining content in one delta, then the calls in order and the
tool_calls finish reason. Argument JSON uses ASCII escapes for a call if its
arguments cannot otherwise be encoded as UTF-8.
Differences from other servers
In vLLM v0.9.2, the non-streaming Hermes parser returns the text before the
first call and drops text between and after calls. It accepts an unclosed final
block if the text through the end decodes to a valid Hermes call object.
Invalid JSON in any block yields the whole text with no calls. Excluding
reasoning from Hermes input requires a configured upstream reasoning parser.
Reasoning controls
How much a model reasons is decided by its chat template. A chat request sets the template's arguments with three fields:
| Field | Meaning |
|---|---|
enable_thinking |
true or false: reasoning on or off for this request |
reasoning_effort |
how much to reason. The Qwen3.8-Flash-Next template uses low, medium and xhigh (its default when thinking is on, and it can think for minutes). The MLX runtime also accepts the names listed below |
chat_template_kwargs |
an object of further template arguments, for example {"enable_thinking": true, "reasoning_effort": "low"} or {"preserve_thinking": false}. On the MLX runtime non-null values of the two fields above win over the same names inside it. A null for any of the three fields, or for enable_thinking / reasoning_effort inside the object, means "not set" |
For Qwen3.8-Flash-Next, reasoning is off on the default MLX launch and in
the shipped oMLX profile. On MLX an accepted effort name other than none, sent alone,
turns it on. On oMLX, send "enable_thinking": true (see "On oMLX" below).
The MLX runtime loads this model through mlx-vlm. The pinned mlx-lm text
loader does not take this model type.
On the MLX runtime
These reasoning, token-limit and max_completion_tokens rules are those of
kamiwaza-mlx-server's own chat endpoint on the MLX runtime.
kamiwaza-mlx-distributed and kamiwaza-mlx-distributed-dflash-server
serve through mlx-lm's own server, which handles these fields its own way.
The API validates the request before the server resolves its tool parser
and reasoning fields. Invalid field types can give a 422, and an
incompatible explicit qwen-xml parser gives a 503. The server checks
template argument names before checking the following combinations.
enable_thinkingtrue withreasoning_effortnonegives a400.enable_thinkingfalse with any effort other thannonegives a400.enable_thinkingtrue and noreasoning_effort: an effort is added only for a model recorded as needing one. Qwen3.8-Flash-Next uses its template default,xhigh.- An effort level and no
enable_thinking: the server turns thinking on only for a model recorded as needing it for the level to take effect. Qwen3.8-Flash-Next reads the effort only while thinking is on.
Rules 1 and 2 hold for every model. What else the launcher does depends on what it knows about the model:
- A model type with recorded reasoning facts. They are in
kamiwaza_mlx/config/reasoning.json, by themodel_typein the model'sconfig.json(for a model named by a hub id, the loaded checkpoint's), and exist only where a model was measured to need them. The server accepts the model's own levels and all six standard OpenAI API effort names (none,minimal,low,medium,highandxhigh). For Qwen3.8-Flash-Next (qwen4_exp), it acceptsnone(enable_thinkingfalse and no level),minimal(sent aslow),low,medium,high(sent asxhigh) andxhigh. After the switch and contradiction checks, an unrecognized effort gives a400listing the accepted names and mappings. A level or mapped name sent alone getsenable_thinkingtrue added (rule 4). Thinking on alone gets nothing added (rule 3). - Any other model type (no recorded facts, or a model whose type cannot
be read). The launcher passes the parsed, merged fields through without
model-specific additions, renames or effort validation. A number is passed
through as an effort too. If either reasoning field is set, the server
adds no default of its own. In that case the server passes the request's
merged template arguments and adds no reasoning default. On a text launch
that renders through mlx-lm's tokenizer wrapper, the wrapper supplies a
missing
enable_thinking: true if it recognizes thinking tokens in the tokenizer, false otherwise. A comparison render can identify a failure due to the request's arguments and return a400with the template error (see "Details"). The server logs pass-through once per process. - A launch where the fields would not reach a template. A request that
sets any of the three is a
400. For otherwise valid fields, the error names the field and the launch. This applies to a text launch where the server formats messages manually, and to a vision-language model on a tool parser other thanqwen-xml(another model, or--tool-call-parser openai-json,minimax-m2ornone), whose prompt is built by mlx-vlm's prompt helper, which this server calls without template arguments. (mlx-vlm 0.7.4's helper could be given them; doing so is a change of its own and is not made in this release.) A request for no reasoning is refused there like any other.
| The request sends | Recorded facts (Qwen3.8-Flash-Next) | No recorded facts | No template gets the fields |
|---|---|---|---|
| none of the three | thinking off on the default vision-language launch | enable_thinking false is sent on the native vision-language path (a template that does not read it ignores it); the loader and template default on text |
the launch's default |
enable_thinking true |
reasoning on at the template's default level; nothing added | to the template as sent | 400 |
enable_thinking false |
reasoning off | to the template as sent | 400 |
reasoning_effort low, medium or xhigh, alone |
reasoning on at that level (enable_thinking true added) |
the server adds no default. Only the effort reaches the native vision-language template. On text, mlx-lm also sends enable_thinking, true for a tokenizer with thinking tokens and false otherwise |
400 |
the same with enable_thinking true |
reasoning on at that level | to the template as sent | 400 |
reasoning_effort minimal or high, alone or with enable_thinking true |
reasoning on at low or xhigh, respectively |
the effort passes through. On text, mlx-lm supplies a missing enable_thinking |
400 |
reasoning_effort none, alone or with enable_thinking false |
reasoning off | the launcher passes the fields through. The template can accept, ignore or refuse them. To ask for thinking off, send enable_thinking false alone |
400 |
any other effort (off, turbo, a number), alone or with enable_thinking true |
400 listing the accepted names and mappings |
the effort passes through. On text, mlx-lm supplies a missing enable_thinking |
400 |
enable_thinking true with none, or enable_thinking false with any effort other than none |
400 that names the contradiction |
the same 400 |
the same 400 |
chat_template_kwargs |
permitted variables go to the template. Nested reasoning controls follow the same rules as the fields | permitted variables go to the template as sent. Unrelated variables alone keep the native vision-language default | 400 if any variable is set. An empty object or one containing only null reasoning controls sets nothing. A null ordinary variable is set |
On the native vision-language path, the server adds its thinking-off
launch default (enable_thinking false) only when the request sets neither
enable_thinking nor reasoning_effort, at the top level or inside
chat_template_kwargs. A recorded effort mapping can also send
enable_thinking false (none on Qwen3.8-Flash-Next). Unrelated template
arguments keep that default. For a text model, mlx-lm 0.31.3 supplies
its loader default, true for a tokenizer with
thinking tokens and false otherwise.
Details:
- What the server cannot see. A template takes what it reads: one that has
no
reasoning_effortrenders the same at every level, and one that has noenable_thinkingrenders the same either way. The server cannot tell whether the template used an argument its renderer accepted. The MiniMax check establishes that the template prints a JSON declaration for every supplied tool in its own<tool>...</tool>block, under the tool's name. It does not read the template's instructions for how to call a tool: a template that prints such declarations and asks for another call format passes, and its calls come back as text. A variable the client sends inchat_template_kwargsis its instruction to the template. Qwen probes with marked tools before the real render, while MiniMax probes after a successful real render. The native Qwen check requires every marker it adds to the names and schemas to appear in the probe, and each real name to occur in the real prompt in literal or JSON-escaped form. If the probe passes, a tool that a template variable removes from the real declarations can still pass the name check when its name occurs elsewhere in the real prompt, such as in the conversation, another tool's name or the forced-choice instruction. Other spellings some clients use (reasoning,thinking,think) are not fields of this API and are ignored like any unknown field. - Rules 1 and 2 read
enable_thinkingastrueorfalse. Only the effortnonemeans no reasoning, ignoring case and surrounding whitespace. Recorded levels and mapped names are matched the same way. The request's ownenable_thinkingfield is a boolean of the API on every tier: it is read as one when the request is parsed (a JSON1or"yes"arrives astrue), before any of this applies. A non-null top-level switch overrides a nested switch. If the nested value survives, Qwen3.8-Flash-Next requires a boolean or returns a400. With no recorded facts its type is preserved. - The server resolves the arguments once per request. Tool checks can
render the conversation more than once. On the native path a recognized
conversation refusal gives a
422before any comparison. For other failures with request arguments, the server tries the render without them. If it succeeds, the server discards that prompt and returns a400naming the fields and quoting up to 300 characters of the template error. On the native vision-language path it reports the thinking-off default if it supplied it to the renderer with unrelated template arguments. If recorded facts changed the arguments, it shows up to 300 characters of the resolved arguments. If the comparison also fails, the server handles its error as it would without the fields. A recognized native conversation refusal gives a422. ATypeErroron a MiniMax text render with tools alone triggers explicit instructions with the request's arguments. For otherTypeError,ValueErroror Jinja template errors, both native and text paths return a503with the exception type and the first 300 characters of its message. Other exceptions propagate. - With
minimax-m2on a text template that takes tools, the server first renders an unforced request with the real tool names. After that succeeds, it renders the same conversation and arguments again with a unique identifier for each tool name. The server uses the real prompt only if the second render succeeds, contains every identifier and exactly reproduces the real prompt when the original names are put back. In the renamed render each name must be the name of a JSON declaration inside its own<tool>...</tool>block. Otherwise it renders explicit instructions with the real tools and the request's arguments. The parser reads only MiniMax's call format, so a template that presents tools another way (Qwen3-Coder's, for example) gets explicit instructions. A forcedtool_choicealways uses explicit instructions listing every supplied tool in request order, then directing the model to call the selected function. MiniMax also uses explicit instructions on vision launches and text launches without template tools.openai-jsonalways uses instructions. - Explicit instructions list every supplied tool in request order,
even when
tool_choiceselects one function. Changing this list between turns can shorten the reusable prefix or leave no usable vision checkpoint. When tools are enabled and the request supplies a nonemptytoolslist, the selected tool must be in that list.openai-jsonaccepts only functions and lists them before the forced instruction. The server returns a400if it cannot prepare the instructions, if they omit a supplied tool or if their text is missing from the final prompt (differences in whitespace are accepted). Settool_choicetononeor launch with--tool-call-parser noneto turn tools off. chat_template_kwargsnames must be nonempty. The launcher reserves names for renderer parameters, includingtools,messages,tokenize,add_generation_promptandchat_template. It also reserves the named parameters found in the renderer signature it inspects, except thatenable_thinkingandreasoning_effortare always allowed as names. Setting a reserved name gives a400.
On oMLX
On oMLX the fields are oMLX's own. The launcher passes the request through unchanged, so what they do is upstream's behavior, and it is not the MLX runtime's (oMLX 0.7.3, Qwen3.8-Flash-Next):
| The request sends | MLX runtime | oMLX |
|---|---|---|
reasoning_effort at a native level, alone |
reasoning on at that level | the effort is a template argument and nothing more: it does not turn reasoning on. With the shipped profile's default of thinking off, nothing changes (measured: 0 reasoning tokens for low, medium and xhigh sent alone). Send "enable_thinking": true with the effort |
reasoning_effort none |
reasoning off alone or with enable_thinking false, 400 with true |
not an off switch: reasoning stays as enable_thinking or the default has it, and with reasoning on oMLX retries none as low (read in oMLX's source, not measured by us). Send "enable_thinking": false |
an effort other than a native level or none |
minimal is sent as low, high as xhigh. Other values give a 400 |
retried under oMLX's own nearest name and, if the template refuses that too, rendered without the effort, with no error (read in oMLX's source, not measured by us) |
the same name as a field and inside chat_template_kwargs |
a non-null top-level field wins | for reasoning_effort, the one inside chat_template_kwargs wins (read in oMLX's source) |
For the same reasoning setting on both runtimes, send enable_thinking
true with a common native level (low, medium or xhigh), or send
enable_thinking false alone for off. Keep top-level and nested fields
consistent.
Measured on Qwen3.8-Flash-Next (E5 build, one 150-word writing prompt, temperature 0):
| Request | MLX runtime with the MTP drafter | oMLX |
|---|---|---|
| thinking off (the default) | 0 reasoning tokens, 4 s | 0, 5 s |
enable_thinking true, reasoning_effort low |
2,250 reasoning tokens, 37 s | 1,059, 35 s |
the same, medium |
2,809, 44 s | 2,218, 66 s |
the same, xhigh (also what thinking on without an effort gives) |
1,821, 28 s | 1,395, 42 s |
reasoning_effort low, medium or xhigh alone |
the same prompt as with enable_thinking true, so the three rows above |
0: reasoning stays off |
Each level gave a different answer on this prompt. On both runtimes
xhigh used fewer reasoning tokens than medium. The effort sets no
separate reasoning budget. A positive max_tokens caps the whole answer.
Qwen3.8-Flash-Next with a drafter
Qwen3.8-Flash-Next has a multi-token-prediction (MTP) head. In the tests
below, its drafter sped up generation and matched plain decoding at
temperature 0. Set a positive max_tokens when comparing outputs.
Which runtime can use the head depends on where it is:
| The head is | Runtime | How |
|---|---|---|
a separate drafter directory (python -m mlx_vlm.split_mtp, model type qwen4_exp_mtp), the usual case for MLX conversions, which strip it from the checkpoint |
MLX | --draft-model <drafter directory>. The command below selects the MLX runtime and prints runtime=mlx (auto: model_type=qwen4_exp, --draft-model given, mlx only) |
| embedded in the checkpoint | oMLX | KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. Refused if checkpoint metadata lists no matching tensor name (see "What a launch on oMLX does"). Not measured by us: no such checkpoint was at hand |
The command for the first case, with the n-gram table kept on disk:
kamiwaza-mlx-server -m ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16 \
--draft-model ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16-mtp \
--ple-store ./e5-table.json
Measured through the server on a Mac Studio (M3 Ultra, 512 GB), E5 build, table on disk, temperature 0, generation rate as the server logs it:
| Configuration | Prose | Code | Accepted drafts (prose / code) | Memory |
|---|---|---|---|---|
| oMLX, no drafter | 38 tok/s | 38 tok/s | 91 GB | |
| MLX runtime, no drafter | 33 tok/s | 33 tok/s | 90 GB | |
| MLX runtime, MTP drafter, default block (4) | 53 tok/s | 73 tok/s | 55% / 88% | 92 GB |
the same, --draft-block-size 3 |
51 tok/s | 64 tok/s | 64% / 92% | 92 GB |
the same, --draft-block-size 5 |
52 tok/s | 78 tok/s | 46% / 82% | 92 GB |
the same, --draft-block-size 8 |
39 tok/s | 74 tok/s | 27% / 65% | 92 GB |
-
The output was the same text with and without the drafter, and at every block size, on all four test prompts (prose, code, and two tool-call turns with 25 tools).
-
The table stays on disk with the drafter on (the memory column).
-
Block size. The drafter's
config.jsonsaysblock_size: 2: that is the depth the head was trained at (one extra token). At run time mlx-vlm applies the head repeatedly, up to a block of 4 for this drafter when--draft-block-sizeis not given. Set at least 2 to allow one target token and at least one draft token per round. The block is a ceiling, not a fixed size: mlx-vlm drafts at the trained depth for the first rounds of a generation and goes up to the block only while at least 65% of recent rounds had their first draft accepted. 5 was a little faster on code and no faster on prose; 8 lost on prose, where most of a long draft is rejected. The default is a reasonable choice for mixed work. -
Before the MLX runtime supported vision prompt reuse, a Mac Studio run on commit
a05a122with the Flash-Next E5 build took about 13–15 s per short-answer tool-calling turn with the MTP drafter. MLX kept the weights loaded and evaluated the full prompt (about 13,000 tokens, 25 tools) on each turn. On warm short-answer turns, oMLX without a drafter kept the prefix and answered in about 1–5 s. Long answers took longer on both runtimes. -
On development commit
9174b27, the same Mac Studio and Flash-Next E5 build (n-gram table on disk) ran a scripted tool-calling conversation of ten steps with a long system prompt and 25 tools. The prompt grew from 13,691 to 18,143 tokens. At each step we timed two complete responses at temperature 0 without streaming. The second request repeated the first request's prompt, so with reuse on it could restore a checkpoint the first request had just stored. At steps 3 and 7 the prompt crossed a boundary. With reuse on, the second request restored 2,048 more tokens and finished sooner than the one-token request at both steps. The first two rows give the minimum / median / maximum for steps 2 to 10. We measured process footprint at the end of each configuration, after the scripted conversation for reuse off and after the additional tool-call phases for reuse on.Reuse off, no drafter Reuse on, no drafter Reuse off, MTP drafter Reuse on, MTP drafter One-token request (taken as time to first token) 14.1 / 15.8 / 17.9 s 1.06 / 1.95 / 2.74 s 13.7 / 15.6 / 17.6 s 1.03 / 1.92 / 2.71 s The same prompt again, up to 400 tokens 14.4 / 16.4 / 18.4 s 1.17 / 2.30 / 2.67 s 14.2 / 16.1 / 18.1 s 1.00 / 1.93 / 2.43 s First prompt of the conversation, one token 62.0 s, including the first read of the weights 16.2 s 13.6 s 13.7 s Prompt tokens restored none 12,288, then 14,336, then 16,384 none 12,288, then 14,336, then 16,384 Final process footprint 91 GB 91 GB 93 GB 93 GB The assistant messages matched with reuse on and off at all ten steps for both requests, without the drafter and with it (40 comparisons). They also matched with and without the drafter. Both comparisons left out the tool-call IDs the server generates per response. The first request in each configuration started with no checkpoint and read the full prompt. With reuse on, each later request of this conversation read on from the last kept 2,048-token boundary. The two checkpoints kept took about 0.5 GB each. Reuse depends on matching prompt tokens. A new conversation can share a kept prefix, and an edit to the history can leave no matching checkpoint. Commit
9174b27also returned a tool call written after prose as a call. Every answer compared here was plain text without a tool call, and the prompts were scripted, with no answer fed back, so that change did not shape the messages compared. The times were taken with that commit's parser in the response path. -
On commit
a05a122, oMLX returned valid calls at its server defaults on all 60 repetitions of the short two-tool weather request, 30 without streaming and 30 with streaming. At the launcher's old MLX defaults (temperature=1.0,top_p=1.0, no top-k), MLX made 59 valid calls without a drafter and 54 with the MTP drafter. Sending"top_p": 0.95brought the drafted run to 60 of 60. The code test sent ten prompts twice each, with 25 tools and about 13,000 input tokens per request. oMLX returned 20 valid calls. MLX returned 19 without a drafter and 18 with the MTP drafter at the launcher's old defaults. The MTP run withtop_p=0.95also returned 18. All 25-tool misses came from one request sent twice per run. On each miss the model answered with a sentence followed by the call. These requests used automatic tool choice, so the server returned that answer as text. In the separate temperature-zero probes, every tool-call turn tested on oMLX without a drafter and on MLX without a drafter or with the MTP drafter returned a valid call. -
On the MLX runtime, ordinary streaming sends reasoning in
content. A generation limit can end the answer before</think>. The--strip-thinkingoption orstrip_thinking: truecan filter paired tags and their contents. With auto-detection on, one-piece answers, text streams and vision-language streams start inside a think block if the request's rendered prompt ends with an open<think>tag, and outside one if it ends with</think>. Think tags earlier in the prompt are not read. Some renderers return the last message itself as the prompt (mlx-vlm's helper does for PaliGemma, Florence-2 and Falcon OCR); when the prompt ends with the text of the request's last message (where the message keeps its parts, its last text part that is not blank) and that text ends with the tag after text of its own, the tag is treated as the client's text and is not read either (a heuristic: the renderer does not say who wrote the end of the prompt). A prompt that ends with neither uses the process-wide detection result, which includes the start-up probe of the template rendered without request arguments. When a block was inferred from the prompt and the output has no think tag at all, an output of 50 tokens or fewer is withheld and a longer one is returned as the answer. On a stream without tools, a block whose opening tag the filter recognizes and that does not close is not sent incontent. A one-piece answer (non-streamed, or streamed with tools) is split by the same rules as before: when its first think tag is</think>, the text after that tag, without leading newlines, is the content, think tags included; otherwise paired blocks are removed, and an unmatched tag and the text after it can stay incontent. The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model. With tools enabled the runtime generates the answer in one piece and returns any extracted reasoning asreasoning_content. oMLX sends reasoning asreasoning_content. -
A DFlash 2 drafter for this model (
incoai/Qwen3.8-Flash-Next-DFlash2, revisiona22a1744) loads with mlx-vlm 0.7.4, but drafting fails on the model's hidden-state layout. The start-up self-test requires a completed draft round and exits 2 with mlx-vlm's error when generation raises. oMLX's DFlash does not cover this model type either.
Native speculative decoding (MTP)
Vision-language checkpoints loaded through mlx-vlm can use mlx-vlm's own
speculative decoding with a separate drafter: a model's native MTP
(multi-token prediction) head, a DFlash2 drafter, or an EAGLE-3 drafter.
In greedy tests with the same positive token limit, Qwen3.8-Flash-Next
produced the same output with and without its MTP drafter.
Many MLX conversions strip the MTP head, so extract it from a checkpoint
that still carries it:
python -m mlx_vlm.split_mtp --model ./Qwen3.8-Flash-Next-FP8 --output ./Qwen3.8-Flash-Next-MTP
python -m kamiwaza_mlx.server \
-m ./Qwen3.8-Flash-Next-mlx \
--draft-model ./Qwen3.8-Flash-Next-MTP
| Flag | Env var | Default | Meaning |
|---|---|---|---|
--draft-model |
KAMIWAZA_MLX_DRAFT_MODEL |
off | Drafter path or HF id. |
--draft-kind |
KAMIWAZA_MLX_DRAFT_KIND |
auto |
auto, mtp, dflash or eagle3; auto reads the drafter's config. |
--draft-block-size |
KAMIWAZA_MLX_DRAFT_BLOCK_SIZE |
mlx-vlm's default for the drafter (4 for the Qwen3.8-Flash-Next MTP head) | Maximum tokens per round, including one target token. The minimum is 2. With 1, mlx-vlm drafts nothing and ends every answer after its first token. The Qwen3.8-Flash-Next MTP head uses shorter rounds when too few recent rounds accept the first draft token. |
--draft-queue-timeout |
KAMIWAZA_MLX_DRAFT_QUEUE_TIMEOUT |
600 |
Seconds a request waits for the drafter before failing with 503 and Retry-After: 5; 0 waits indefinitely. The wait ends when the request takes the drafter. Parsing the request and checking its token limit and reasoning fields come before the wait; loading its images and building its prompt come after. |
--draft-stream-send-timeout |
KAMIWAZA_MLX_DRAFT_STREAM_SEND_TIMEOUT |
60 |
Seconds a draft-enabled stream may wait for the client to accept one chunk before it is aborted and the drafter released; 0 disables. |
- With a native drafter, an unset, null, zero or negative token limit becomes
the served context window minus the token count of the prompt text, with a
minimum of 1. Image tokens are excluded, so an image request can get more
room than the window has left. If the window is unknown, the limit is
effectively unbounded (
2**31 - 1). A positive limit and every limit on a launch without a native drafter go to generation as before. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens. On thekamiwaza-mlx-serverchat endpoint on the MLX runtime,max_tokensandmax_completion_tokensname this same limit. Send either or both when equal. Null for either means unset. Different integers give a400, for examplemax_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields.The vision path does not reportfinish_reason: length, so a text answer that reaches the substituted limit reportsstop. - Startup fails if the drafter cannot pair with the target model, if the model
loads through
mlx-lm(for example with--text-only), or if--dflash-draft-modelis also set. The server starts serving only after the requested drafter completes a round in its self-test. - The drafter serves one generation at a time, so draft-enabled requests
queue. A streamed response holds the drafter until its last chunk is sent,
so a slow reader delays everyone behind it (each chunk must be accepted
within
--draft-stream-send-timeout, or the stream is aborted).--draft-queue-timeoutbounds the wait with a503, but the timer runs on the server's event loop: while a non-streamed or tool-call generation is running it cannot fire, so a waiter may be answered (or rejected) only after that generation finishes. A client that disconnects while queued is skipped when its turn comes, on a best-effort basis (a disconnect the server has not yet read is not detected). A request that completes a draft round logs the mean tokens per round (one target token plus the accepted drafts) and the fraction of drafted tokens accepted. - Startup also fails if the hidden size or vocabulary size the launcher
reads from the drafter's configuration (its text configuration first,
then the top level) differs from the target model's (for example a
drafter built for another model family; a drafter that records the
target's width as
backbone_hidden_sizeortarget_hidden_sizeis compared by that value; sizes recorded elsewhere, such as an EAGLE-3 drafter's vocabularies, are not compared), or if an explicit--draft-kinddisagrees with the drafter's own type. - After loading the drafter, the server tries up to three greedy text
generations of up to eight tokens each. A generation passes when it
finishes with at least one completed draft round, even if the target
rejected every proposal.
If generation raises or all three finish without a round, the launch
exits 2 with the reason. The test warms the weights its prompts use.
The log line starting with
Drafter self-test passed inreports the elapsed time and completed rounds. The earlier single-generation test took 3 to 13 s for Qwen3.8-Flash-Next in our runs, depending on the system's file cache. Sampled decoding, image inputs and the HTTP layer are outside this test. - Successful non-streamed responses for requests that complete a draft round include
X-Speculative-Tokens-Per-Round,X-Speculative-Acceptance-RateandX-Speculative-Rounds. A request limited to one token has none of these headers. - Measured through the server on Qwen3.8-Flash-Next with its MTP head (M5 Max, 128 GB, temperature 0): 37 tok/s without a drafter; 52 tok/s on prose and 73 tok/s on code with it. Gains are smaller with sampling (temperature 1.0: 36 and 62 tok/s).
N-gram table on disk (Qwen3.8-Flash-Next)
Qwen3.8-Flash-Next (mlx-vlm model type qwen4_exp) carries a very large
per-layer n-gram embedding table, also called the PLE or "engram" table, of
about 51 billion values. It is a lookup table. Loaded as ordinary
parameters it occupies tens of gigabytes of unified memory; mlx-vlm can
instead memory-map the table inside the checkpoint's safetensors files and read
only the rows each token needs.
A model directory opts in through text_config.ple_storage in its
config.json (mlx-vlm's prepare_external_ple_model writes such a directory).
--ple-store makes the same choice at launch, without preparing a second
directory, and can point at the table of another conversion of the same model:
# Index the table of a converted checkpoint in place (writes a small JSON file;
# no weights are copied). The manifest refers to the checkpoint by relative
# path, so keep the two where they are.
python -m kamiwaza_mlx.ple_store ./Qwen3.8-Flash-Next-mlx-8bit ./tables/q8/ple-store.json
# Serve 4-bit weights with that 8-bit table read from disk.
python -m kamiwaza_mlx.server \
-m ./Qwen3.8-Flash-Next-mlx-4bit \
--ple-store ./tables/q8/ple-store.json
| Flag | Env var | Default | Meaning |
|---|---|---|---|
--ple-store |
KAMIWAZA_MLX_PLE_STORE |
off | Manifest of the n-gram table to read from disk. Replaces the table in the model directory, and a manifest the model's config.json declares. |
- The table bundled in the model directory is not loaded when
--ple-storeis set. To keep a model's own table on disk, index that same directory and pass the manifest. - Use a table from a conversion of the same model. The launcher can only
check its shape: a table with a different number of rows or row width is
refused, but a table of the same shape from another model is not detected
and would give wrong output.
Supported table layouts are 4-bit and 8-bit affine with group size 32, and
4-bit
nvfp4. Anything else fails at startup with the reason. - Startup fails, and the server never serves without the requested table, if
the manifest is missing or incomplete, if the model is not a
qwen4_expmodel, if--text-onlyor--dflash-draft-modelis also set, if the table does not fit the model, or if the installedmlx-vlmhas no on-disk table support (the pinned 0.7.4 has it). A table that does not fit is reported asthe n-gram table of --ple-store <manifest> could not be used: <reason>; a failure of the model itself keeps its own message. - The model must be one this server loads through
mlx-vlm, which is what it does for a conversion that keeps its vision tower. For any other model startup fails; it is not forced throughmlx-vlm. - Lookups read from the page cache, so the first tokens after a cold start are
slower.
--cache-rows Non the manifest command records a row cache size in the manifest: the N most recently used rows are kept in memory. With--ple-storethe manifest's value is the one used, whatever the model directory's ownconfig.jsondeclares.
Experimental DFlash speculative decoding
DFlash is available as an opt-in text-only generation path. Configure a draft
model and the server will use a DFlash speculative loop instead of the normal
mlx_lm.generate.stream_generate loop. The default backend is auto: it uses
the standalone dflash-mlx implementation when installed, otherwise it falls
back to ZLab's dflash.model_mlx.
With --strip-thinking or strip_thinking: true, the stream of this path
decides per request whether an answer starts inside a think block (see
--strip-thinking under "Most useful flags"). That rule is covered by unit
tests only on this path; it was not run with a live model.
The DFlash backends install from git (PyPI does not allow direct-URL dependencies in package metadata). These are the tested pins:
# ZLab backend
pip install "dflash[mlx] @ git+https://github.com/z-lab/dflash.git@6d6229eaddce58fcd7e4cc91945da632f5544c86"
# Standalone dflash-mlx backend
pip install "dflash-mlx @ git+https://github.com/bstnxbt/dflash-mlx.git@fada1eb2b75cd1c875ca6547b6518783fd3d2956"
python -m kamiwaza_mlx.server \
-m mlx-community/Qwen3.5-27B-5bit \
--dflash-draft-model z-lab/Qwen3.5-27B-DFlash \
--dflash-backend auto \
--dflash-block-size 16 \
--dflash-default-max-tokens 2048 \
--dflash-max-context -1
The first five environment variables correspond to the DFlash options above. The last three explicitly set verification and stream synchronization to their defaults.
export KAMIWAZA_MLX_DFLASH_DRAFT_MODEL=z-lab/Qwen3.5-27B-DFlash
export KAMIWAZA_MLX_DFLASH_BACKEND=auto
export KAMIWAZA_MLX_DFLASH_BLOCK_SIZE=16
export KAMIWAZA_MLX_DFLASH_DEFAULT_MAX_TOKENS=2048
export KAMIWAZA_MLX_DFLASH_MAX_CONTEXT=-1
export KAMIWAZA_MLX_DFLASH_VERIFY_MODE=auto
export KAMIWAZA_MLX_DFLASH_VERIFY_QMM=true
export KAMIWAZA_MLX_DFLASH_STREAM_SYNC_EVERY=0
Notes:
- DFlash is only enabled when
--dflash-draft-modelis non-empty, and that forces a text-onlymlx-lmtarget load even for Qwen3.5 conversions that retain avision_config. - The Qwen3.5-27B drafter is trained for target
Qwen/Qwen3.5-27B; MLX quantized conversions such asmlx-community/Qwen3.5-27B-5bitandmlx-community/Qwen3.5-27B-6bitare structurally compatible targets to try, but acceptance and quality need measurement per quant. - The
z-lab/Qwen3.5-27B-DFlashdraft config uses fivefull_attentionlayers. Thedflash-mlxruntime has draft sink/window machinery for other draft shapes, but those knobs do not change this Qwen3.5 speculative model's draft attention. - Server KV and prefix caches are disabled while DFlash is enabled because DFlash allocates its own per-request target/draft prompt caches and mutates target model hooks during generation.
--dflash-backend autopicksdflash-mlxwhen the standalonedflash_mlxpackage is installed (see the tested git pin above); otherwise it falls back to the ZLab backend. The inlinedflash-mlxadapter runs without that package's own prefix snapshot cache, so it is meant for local kernel/runtime experiments, not as a replacement for that package's full OpenAI-compatible server. It also installs a Qwen target-op compatibility patch that skips the full-vocabulary lm-head projection for large prompt-prefill chunks whose logits are discarded by the DFlash loop, and disables unused prefix-snapshot work because the inline adapter does not provide adflash-mlxprefix cache.- For eligible 4-bit or 8-bit MLX target linears, the
dflash-mlxadapter also installs that package's verifier-linear wrappers. Use--dflash-verify-mode offto disable the swap, or--no-dflash-verify-qmmto keep the wrappers while disabling the custom qmm kernels. --dflash-max-contextis a degradation guard: prompts at or above that token count bypass DFlash and use the normal MLX generation path. Omit it, or set it to-1, to use the conservative auto cap of4000prompt tokens. Set it to0only when you intentionally want to route every prompt through DFlash. Because DFlash mode disables the server-level shared KV/prefix caches, bypassed requests do not use shared prefix-cache optimizations, but they still allocate the normal bounded per-request generation cache.- DFlash generation is serialized with an internal lock. That protects the target model hidden-state hooks and Qwen3.5 gated-delta rollback patch.
- Extra
mx.synchronize()calls in the DFlash streaming loop are disabled by default. Set--dflash-stream-sync-every Nonly when diagnosing stream timing or device scheduling.
Note: Not specific to Kamiwaza (that is, you can use on any Mac, Kamiwaza not required)
pip install kamiwaza-mlx
# start the server
a) python -m kamiwaza_mlx.server -m ./path/to/model --port 18000
# or, if you enabled the optional entry-points during install
b) kamiwaza-mlx-server -m ./path/to/model --port 18000
# chat from another terminal (note: specify --host to match server port)
python -m kamiwaza_mlx.infer --host localhost:18000 -p "Say hello"
The remainder of this README documents the original features in more detail.
MLX-LM 🦙 — Drop-in OpenAI-style API for any local MLX model
A FastAPI micro-server (server.py) that speaks the OpenAI
/v1/chat/completions dialect, plus a tiny CLI client
(infer.py) for quick experiments.
Ideal for poking at huge models like Dracarys-72B on an
M4-Max/Studio, hacking on prompts, or piping the output straight into
other tools that already understand the OpenAI schema.
✨ Highlight reel
| Feature | Details |
|---|---|
| 🔌 OpenAI compatible | Same request / response JSON (streaming too) – just change the base-URL. |
| 📦 Zero-config | Point at a local folder or HuggingFace repo (-m /path/to/model). |
| 🖼️ Vision-ready | Accepts {"type":"image_url", …} parts & base64 URLs – works with Qwen-VL & friends. |
| 🎥 Video-aware | Auto-extracts N key-frames with ffmpeg and feeds them as images. |
| 🧮 Usage metrics | Prompt / completion tokens + tokens-per-second in every response. |
| ⚙️ CLI playground | infer.py gives you a REPL with reset (Ctrl-N), verbose mode, max-token flag… |
🚀 Running the server
# minimal
python -m kamiwaza_mlx.server -m /var/tmp/models/mlx-community/Dracarys2-72B-Instruct-4bit
# custom port / host
python -m kamiwaza_mlx.server -m ./Qwen2.5-VL-72B-Instruct-6bit --host 0.0.0.0 --port 12345
Default host/port: 0.0.0.0:18000
Most useful flags:
| Flag | Default | What it does |
|---|---|---|
-m / --model |
mlx-community/Qwen2-VL-2B-Instruct-4bit |
Path or HF repo. |
--host |
0.0.0.0 |
Network interface to bind to. |
--port |
18000 |
TCP port to listen on. |
-V / --vision |
off | Force vision pipeline; otherwise detect declared vision config. |
--text-only |
off | Explicit language-only load for converted checkpoints retaining vision metadata; mutually exclusive with --vision. |
--strip-thinking |
off | Removes <think>…</think> blocks from model output. A request can set strip_thinking: true. With auto-detection on, one-piece answers, text streams and vision-language streams start inside a think block if the request's rendered prompt ends with an open <think> tag, and outside one if it ends with </think>. Think tags earlier in the prompt are not read, nor a tag that ends the text of the request's last message (where the message keeps its parts, its last text part that is not blank) where the prompt ends with that text and the text has more of its own before the tag (a heuristic for renderers that return the last message as the prompt). A prompt that ends with neither uses the process-wide detection result, which includes the start-up probe of the template rendered without request arguments. The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model. |
--draft-model |
off | Native mlx-vlm speculative drafter (MTP, DFlash2, EAGLE-3); see Native speculative decoding. |
--vlm-prompt-reuse |
auto |
Reuse matching prompt prefixes on eligible MLX vision-language models for requests without images or media placeholders. See Prompt reuse for vision-language models. |
--sampling-defaults |
model |
Choose defaults for sampling fields omitted from a request. See Sampling defaults. |
--enable-prefix-caching |
True |
Enable automatic prompt caching for text-only models. If enabled, the server attempts to load a cache from a model-specific file in --prompt-cache-dir. If not found, it creates one from the first processed prompt and saves it. |
--prompt-cache-dir |
./.cache/mlx_prompt_caches/ |
Directory to store/load automatic prompt cache files. Cache filenames are derived from the model name. |
KV cache flags (all KV-related CLI knobs)
System-prefix cache (system-only)
| Flag | Default | What it does |
|---|---|---|
--enable-prefix-caching |
True |
Enable system-prefix caching for text-only models. |
--prompt-cache-dir |
./.cache/mlx_prompt_caches/ |
Directory for system-prefix cache files (.safetensors, .len, .hash). |
--system-cache-max-tokens |
2048 |
Max tokens to cache from the system prompt (0 = unlimited). If the system prompt exceeds this cap, system caching is skipped. |
--prefix-cache-headroom |
64 |
Extra tokens reserved beyond the system prompt length when sizing the system cache. |
Conversation cache (simple global prefix-match)
Simple mode uses a single text-model conversation cache. If the previous prompt is a prefix of the incoming prompt, the server skips prefill for that prefix. Otherwise it resets the cache. Conversation IDs appear in logs and metadata.
| Flag | Default | What it does |
|---|---|---|
--disable-kv-cache |
off | Disable the launcher's retained KV caches, including vision prompt checkpoints. Generation still uses temporary KV state within each request. |
--kv-cache-max-tokens |
0 |
Per-cache upper bound (0 = min(model context, 128k)). |
--kv-cache-keep |
4 |
Tokens to keep when trimming rotating caches. |
--kv-cache-idle-release-seconds |
0 |
Release KV caches, including retained vision prompt checkpoints, on the next chat request after the idle interval (0 disables idle release). |
--kv-cache-hard-reserve |
True |
Fully reserve KV memory up to sizing target. |
--kv-cache-warmup |
False |
Run a warm-up pass at startup to materialize KV shapes. |
--kv-cache-warmup-tokens |
0 |
Warmup tokens (0 = use resolved target size). |
--retain-mx-cache |
False |
Keep MX allocator memory (disables mx.clear_cache). |
Prompt reuse for vision-language models
mlx-vlm can save the target model's prompt state at a 2,048-token boundary. An eligible request can continue from the longest retained prefix that matches token for token. Prompts of 2,048 tokens or fewer are always read in full. For 2,049 tokens, the end-minus-one checkpoint falls exactly at 2,048 and can be kept. The alignment check rejects an end-minus-one checkpoint only when it falls off the grid. Even an identical prompt leaves its final token to read.
Requests with images or a media placeholder such as <|image_pad|> anywhere
in the rendered prompt bypass reuse. Either can cause mlx-vlm 0.7.4 to split
a prefill chunk at a different token, even on a cache miss. When the input
IDs contain no media placeholders, chunked prefill uses mlx-vlm's default
step, and the request uses MTP or no drafter, restoring a grid boundary
preserves the forward calls for the remaining prompt.
The offline Metal test compares generated tokens with
and without reuse at temperature 0 (greedy decoding) on tiny random Qwen
weights for repeated prompts, growing conversations and shared-prefix forks.
Each running request that restores a checkpoint uses its own copy. Reuse adds no new request-wide lock. The manager takes its own lock around storing and looking up checkpoints.
| Flag | Environment variable | Default | Meaning |
|---|---|---|---|
--vlm-prompt-reuse {auto,on,off} |
KAMIWAZA_MLX_VLM_PROMPT_REUSE |
auto |
Enable reuse when the startup checks pass. Use on to require the manager at startup or off to disable it. |
--vlm-prompt-reuse-checkpoints N |
KAMIWAZA_MLX_VLM_PROMPT_REUSE_CHECKPOINTS |
2 |
Keep at most N checkpoints in memory across requests. The minimum is 2 because mlx-vlm reserves one slot when choosing up to N−1 intermediate boundaries per request. |
--vlm-prompt-reuse-max-gb G |
KAMIWAZA_MLX_VLM_PROMPT_REUSE_MAX_GB |
8 |
Maximum total size of retained checkpoint cache buffers in GiB. Token lists and bookkeeping are outside this limit. The value converted to bytes must be finite and at least one byte. |
Blank or whitespace-only environment values use the defaults above. The MLX
runtime validates both numeric settings at startup in every mode, including
off. Reuse requires KV caching and a listed vision-language model type.
The list currently contains only qwen4_exp (Qwen3.8-Flash-Next). See
config/prompt_reuse.json.
The eligibility check runs after the model and any vision drafter have loaded
and the drafter's self-test has passed. It accepts MTP or no drafter. DFlash
and EAGLE-3 need hidden states for the entire prompt. The launcher verifies
that the tokenizer resolves each configured media token ID to a placeholder
string and that tokenizing a plain, media-free probe produces none of those
IDs. It builds the manager, checks the whole-model checkpoint plan and the
model's cache layout, and exercises the manager's statistics, byte accounting
and clear operations. If reuse is unavailable, auto leaves it off and
on exits 2. Starting successfully with on guarantees that these
checks passed. A checkpoint can still be too large to retain. A cap of one
byte stores no checkpoints.
Conversation IDs and reset_conversation apply to the text-model
conversation cache. The first vision request in a new conversation can restore
a kept prefix if its tokens match, for example when it starts with the same
system prompt and tools. This also applies with reset_conversation: true.
If an edit to the history leaves no matching checkpoint, the next request
reads the full prompt. To force the vision model to read every prompt in
full, start the MLX runtime with --vlm-prompt-reuse off.
The launcher decides whether to create the manager and keeps it in memory.
It constructs the manager directly, bypassing the mlx-vlm factory that
reads APC_ENABLED, APC_DISK_ENABLED, APC_DISK_PATH, APC_BLOCK_SIZE
and APC_NUM_BLOCKS. These variables have no effect on vision prompt reuse.
The launcher supplies the settings for APC_CHECKPOINT_ENTRIES and APC_MEMORY_MAX_GB
from the two numeric options. It sets the guard (APC_CHECKPOINT_GUARD_TOKENS) to
1 token, the interval (APC_CHECKPOINT_INTERVAL_TOKENS) to the default
prefill step and the reserve (APC_MEMORY_RESERVE_GB) to mlx-vlm's automatic
value. The manager reads some APC_* settings directly. For example, the
launcher caps a valid APC_EXACT_MIN_TOKENS at one prefill step. A
nonnumeric value makes the startup check fail. The launcher reports mlx-vlm's
error text and appends the names of inherited APC_* variables. The error
text itself can include a value.
Retained checkpoints are evicted least recently used first. Under memory pressure mlx-vlm can also evict a checkpoint or skip a store or restore. Even for a single Qwen request, mlx-vlm makes room before extracting a checkpoint from the prompt cache. Under pressure it can evict a matching prefix at that step, even if the launcher then declines to store an off-grid end-of-prompt checkpoint. If all matching prefixes were evicted, the next replay reads the whole prompt.
A successful non-streamed vision response reports the restored prefix length
in usage.prompt_tokens_details.cached_tokens, including zero on a miss or
bypass. For a streamed request, the per-request restore count is available
only in the generation log. The X-Conv-* headers describe the text
conversation cache. With reuse enabled, the launcher logs the restored and
total prompt tokens for each eligible generation that produced a result. The
log records outcome=completed, outcome=abandoned or outcome=failed,
even if the generation ended before its first result. The launcher logs a
bypass reason for images or media placeholders. A warning records an increase
in the manager's memory skip or memory eviction counter only when the snapshots
before and after the request both report a nonnegative integer for that counter
(booleans are invalid).
GET /v1/kv_cache/stats reports the startup decision and limits in
vlm_prompt_reuse. checkpoints is the configured entry limit.
resident_bytes is the total size of the kept checkpoints' cache buffers.
It excludes copies in use by running requests and is null when reuse is
off. If the statistics read fails or a required field is missing or is
not a nonnegative integer (booleans are invalid), the block includes
statistics_error, and resident_bytes and
mlx_vlm_checkpoint_counters are both null.
launcher_counterscounts eligible requests and tracks image and media placeholder bypasses separately.restored_requestscounts eligible requests whose generation result reported a restored prefix. If mlx-vlm restores a prefix and generation fails before its first result, the launcher counts the request ineligible_requestsandeligible_without_result. Only mlx-vlm records the restore in that case, so its counters can differ from the launcher's. The launcher's prompt-token and restored-token totals cover eligible requests that produced a result. The launcher keeps its counts for the lifetime of the process.mlx_vlm_checkpoint_countersreports the manager's checkpoint stores and hits, restored tokens, memory skips, memory evictions, evictions and rejects. Ordinary replacement at the entry limit does not increment mlx-vlm's eviction counter. The manager's counters restart at zero when it clears the checkpoints, including on idle release. If the idle clear raises an exception, the launcher warns that checkpoints may have been kept and continues serving the request.
totals.kv_bytes includes the resident checkpoint bytes. It is null
when statistics_error is present, because the checkpoint bytes are then
unknown.
Experimental multi-node via mlx.distributed
This section documents the direct server.py distributed bootstrap path.
It is distinct from the current Kamiwaza-managed two-node serving flow, which
uses kamiwaza-mlx-distributed plus upstream mlx_lm.server and is documented
in MLX_TWO_NODE_README.md.
The server can bootstrap a two-node mesh using mlx.distributed. Set a rendezvous host via PAIRED_HOST (optionally in a .env file) and launch each node with matching ranks/world-size. The helper will automatically read .env files passed via --distributed-env-file or located beside the server script.
# shared settings (either export or place in .env)
PAIRED_HOST=10.0.0.2
PAIRED_PORT=17863
WORLD_SIZE=2
# leader node (rank 0 hosts FastAPI)
RANK=0 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model
# worker node (rank 1 participates in mlx.distributed but does not bind HTTP)
RANK=1 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model
Useful knobs:
--distributed– force-enable/disable distributed mode (auto when a paired host is set, orPAIRED_WORLD_SIZE/MLX_DISTRIBUTED_WORLD_SIZE/--distributed-world-sizeis above 1; a bareWORLD_SIZEdoes not turn it on).--distributed-rank/--distributed-world-size– overrideRANK/WORLD_SIZEenv vars.--distributed-host/--distributed-port– overridePAIRED_HOST/PAIRED_PORT.--distributed-server-rank– choose which rank should host the HTTP server (defaults to 0).
In this direct path, non-leader ranks simply keep the MLX runtime alive for collective ops once the model weights are synchronized. Treat it as a bootstrap/experimentation path, not as the primary Kamiwaza-managed TP=2 serving mode.
💬 Talking to it with the CLI
python kamiwaza_mlx/infer.py --host localhost:18000 --max_new_tokens 2048
Interactive keys
- Ctrl-N: reset conversation
- Ctrl-C: quit
🌐 HTTP API
GET /v1/models
Returns a list with the currently loaded model:
{
"object": "list",
"data": [
{
"id": "Dracarys2-72B-Instruct-4bit",
"object": "model",
"created": 1727389042,
"owned_by": "kamiwaza"
}
]
}
The created field is set when the server starts and mirrors the OpenAI API's timestamp.
POST /v1/chat/completions
{
"model": "Dracarys2-72B-Instruct-4bit",
"messages": [
{ "role": "user",
"content": [
{ "type": "text", "text": "Describe this image." },
{ "type": "image_url",
"image_url": { "url": "data:image/jpeg;base64,..." } }
]
}
],
"max_tokens": 512,
"stream": false
}
On the kamiwaza-mlx-server chat endpoint on the MLX runtime,
max_completion_tokens and max_tokens name the same limit. Send either
field, or both with equal values. A null for either field means it is unset.
Different integers return HTTP 400, for example
max_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields.
With a native drafter, an unset, null, zero or negative limit becomes the
served context window minus the token count of the prompt text, with a minimum
of 1. Image tokens are excluded, so an image request can get more room than
the window has left. An unknown window gives an effectively unbounded limit
of 2**31 - 1. A positive limit and every limit on a launch without a native
drafter go to generation as before. With a native drafter the server treats
zero as unset. On a vision-language launch without one it passes zero
through to mlx-vlm; for an autoregressive model its generation then emits no tokens.
The vision path does not report finish_reason: length, so a text answer
that reaches the substituted limit reports stop.
Response (truncated):
{
"id": "chatcmpl-d4c5…",
"object": "chat.completion",
"created": 1715242800,
"model": "Dracarys2-72B-Instruct-4bit",
"choices": [
{
"index": 0,
"message": { "role": "assistant", "content": "The image shows…" },
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 143,
"completion_tokens": 87,
"total_tokens": 230,
"tokens_per_second": 32.1
}
}
Add "stream": true and you'll get Server-Sent Events chunks followed by
data: [DONE].
System Prefix Caching (Text-Only Models):
- Purpose: Dramatically speed up repeated queries that share the same system context (e.g., large document in
role: system). The server caches only the system message(s), not the whole prompt, so subsequent turns process only new user tokens. - Flags:
--enable-prefix-caching(defaultTrue)--prompt-cache-dir(default./.cache/mlx_prompt_caches/)--system-cache-max-tokens(default2048,0disables the cap)--prefix-cache-headroom(default64)
- How it works (high‑level):
- On first request with a system message, the server builds a KV cache for just the system portion and saves three files under
--prompt-cache-dir:<model>.safetensors(KV),<model>.safetensors.len(token count),<model>.safetensors.hash(SHA256 over token IDs)
- On subsequent requests with the same system text (hash matches), the server deep‑copies the cached KV and processes only new user/assistant tokens.
- If the system message changes, the old cache is discarded and replaced automatically.
- If the system prompt exceeds
--system-cache-max-tokens, system caching is skipped and the full prompt is prefetched normally.
- On first request with a system message, the server builds a KV cache for just the system portion and saves three files under
- Example: A 10,000‑token system document is processed once; later questions only process the user tokens.
- Notes: text‑only models; fully transparent to clients (no special fields needed).
Conversation KV Caching (Long chats, fast follow‑ups):
- Rationale: Reuse KV across turns so only the tail of the prompt is prefetched.
- Behavior:
- Provide
conversationorconversation_id(orX-Conversation-Id) if you want IDs reflected in logs/metadata. If omitted, the server usesdefault. - Successful JSON and SSE responses include the text conversation cache headers:
X-Conv-Id(resolved ID),X-Conv-KV(cache mode),X-Conv-Cached-Tokens,X-Conv-Processing-Tokens.
- Non‑stream JSON also includes
usage.input_tokens_details.cached_tokensandmetadata.conversation_id. - The global cache is capped at
min(model context, 128k)unless overridden by--kv-cache-max-tokens. - Only one request at a time uses the text conversation cache. Concurrent text requests that bypass a busy cache report
X-Conv-KV: busy. Vision requests with KV caching enabled reportX-Conv-KV: vision, including when they restore a prompt checkpoint.
- Provide
- Breaking change: the legacy
/v1/conv_kv/*endpoints were removed in this branch.
🛠️ Internals (two-sentence tour)
- server.py – loads the model with mlx-vlm, converts incoming OpenAI vision messages to the model's chat-template, handles images / video frames, and streams tokens back. For text-only models, if enabled via server flags, it automatically manages a system message cache to speed up processing when multiple queries reference the same system context.
- infer.py – lightweight REPL that keeps conversation context and shows latency / TPS stats.
That's it – drop it in front of any MLX model and start chatting!
Metadata
Release files for kamiwaza-mlx 0.2.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kamiwaza_mlx-0.2.9.tar.gz | 515.9 kB | Without provenance | Details for kamiwaza_mlx-0.2.9.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kamiwaza_mlx-0.2.9-py3-none-any.whl | Python 3 | none | any | Details for kamiwaza_mlx-0.2.9-py3-none-any.whl |
Total release size: 765.1 kB
Release files / kamiwaza_mlx-0.2.9.tar.gz
| Download URL | kamiwaza_mlx-0.2.9.tar.gz |
|---|---|
| Size | 515.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
58fecd0c6f838bcdd7c582811cab459a7ea46d962d09689caef9dfca479a9e84
|
|
BLAKE2b-256 checksum How to use checksums |
866c1630d6c5dcc68da9710f45739e119053fb456185972398f5e2f17571afad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / kamiwaza_mlx-0.2.9-py3-none-any.whl
| Download URL | kamiwaza_mlx-0.2.9-py3-none-any.whl |
|---|---|
| Size | 249.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
42a7db836a3257fdc7a83d55fb36f61b2ac28b758ec44b4ffbb0342d2ba12f41
|
|
BLAKE2b-256 checksum How to use checksums |
78c2ae2e0556d55c6781ceb93736bfb234fcf4ce1022fb3ef6605bf2b960d486
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Security for kamiwaza-mlx 0.2.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.