Skip to main content

Kamiwaza-MLX 📦

A simple openai (chat.completions) compatible mlx server that:

  • Supports both vision models (via flag or declared config detection) and text-only models
  • Supports streaming boolean flag
  • Has a --strip-thinking which will remove tag (in both streaming and not) - good for backwards compat
  • Supports usage to the client in openai style
  • Prints usage on the server side output
  • Appears to deliver reasonably good performance across all paths (streaming/not, vision/not)
  • Has a terminal client that works with the server, which also support syntax like image:/path/to/image.png Describe this image in detail
  • Experimental direct multi-node bootstrap via mlx.distributed when PAIRED_HOST is provided; the current Kamiwaza TP=2 launcher/proxy flow is documented in MLX_TWO_NODE_README.md

Tested largely with Qwen2.5-VL and Qwen3 models.

Current MLX Stack

The Desktop candidate branch locks the MLX compatibility epoch as one dependency unit. Dependency/import checks and offline regressions pass; semantic vision and distributed runtime qualification remain open. These versions must move together because current mlx-vlm requires MLX 0.32 while mlx-lm remains on its 0.31 release line:

  • mlx==0.32.3
  • mlx-lm==0.31.3
  • mlx-metal==0.32.3
  • mlx-vlm==0.7.4
  • mlx-audio==0.5.7

Vision capability is fail-closed. If a model config declares a vision tower, the server loads it through mlx-vlm or fails startup with the underlying diagnostic; it never silently falls back to a language-only mlx-lm process. Repository names alone do not declare vision capability. Use --text-only for an intentionally text-only conversion that retains vision config metadata; it is mutually exclusive with --vision. Requesting DFlash also remains an explicit text-only mode and is logged as such for vision-capable checkpoints.

Earlier-epoch testing of mlx-community/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-4bit used mlx_vlm generate for combined text, image, and audio inputs. This model has not been requalified against the current lock. Native video is not yet functional in this MLX path; the current upstream Nemotron MLX model code raises NotImplementedError: Efficient video sampling is not implemented for Nemotron Omni yet.

Checkpoint code on the MLX runtime

On the MLX runtime of kamiwaza-mlx-server, --trust-remote-code is off by default, including when this entry point runs on multiple ranks. The flag allows checkpoint and processor-requested Python when loading the target and any drafter. Their loaders may also request code from another repository. Each loader resolves Hub IDs as it did before this change. With trust off, the launcher checks the resulting directory before passing it to the loader. Identical Hub IDs share a resolution when they use the same resolver. A fallback from text to vision uses mlx-vlm's resolver and vision check. If each resolver returns a config that selects the other loader, startup asks for the local directory of the intended revision.

Run python -m kamiwaza_mlx.server from a trusted working directory, never from inside a model directory. Python puts the working directory on the import path by default for -m. The installed kamiwaza-mlx-server console script uses its script directory for that import path entry.

Path Refused before load with trust off Why Measured (offline stand-ins, pinned packages)
Text target (mlx-lm), including --text-only model_file in config.json mlx-lm imports the named file even with trust off. A config auto_map passes the directory check. A built-in tokenizer can still load when Transformers declines a custom AutoConfig. In 10 noninteractive llama attempts with the trust argument omitted, no auto_map code ran. The loader imported a declared model_file.
Vision target (mlx-vlm), including a fallback from text model_file in config.json, a non-empty auto_map, audio_tokenizer_class or audio_tokenizer_name_or_path, or a model type listed below mlx-vlm can import the model file or delegate tokenizer and processor loading. The launcher checks before each call to the vision loader. On qwen4_exp, auto_map code ran in 8 of 12 attempts with the trust argument omitted and 0 of 12 with trust_remote_code=False. Both settings imported a declared model_file. On qwen3_vl, tokenizer and config auto_map code ran with the weights step stubbed.
Drafter (--draft-model or --dflash-draft-model) model_file in config.json Two drafter loaders import the named file. A drafter's auto_map alone does not trigger this check. The drafter check runs before the target loads. In 21 loads (seven per backend), dflash-mlx and mlx-vlm load_drafter imported model_file. Z-Lab imported no checkpoint module with Hub lookup redirected to the fixture. None of the three imported auto_map code.

With trust off, all three paths check for model_file at any depth in the config.json the loader opens, including spellings the filesystem treats as equivalent. An unreadable config stops startup, and a drafter directory must contain config.json. Vision also checks for auto_map and the two audio tokenizer keys at any depth in each top-level JSON file. The vision check applies its canonical file rules to its fixed list of loader filenames and the config Transformers selects from configuration_files in config.json. It refuses dangling links and requires each present canonical file to be readable JSON of at most 64 MiB. It requires a JSON object except for engram_token_map.json, which may have any JSON root type. The size limit, JSON syntax check, and vision declaration scan still apply. For other top-level JSON files the vision check skips unreadable or malformed files, warns for oversized files, and refuses excessive nesting. A Python file that nothing names can remain in any directory.

On the text path, a tokenizer auto_map also passes the directory check. The launcher passes the explicit trust setting to mlx-lm's tokenizer and to mlx-vlm's model and native drafter loaders. Opting in passes True and can load a custom text tokenizer without prompting. Processor-only declarations do not trigger the text directory check. With trust off, the Transformers guard refuses remote-only classes and direct dynamic class imports, even if a dependency asks for trust_remote_code=True. A loader that recovers using built-in code can finish loading.

On refusal, the launcher exits 2 and attempts to write one error line. A declaration refusal lists the file and matching fields, shortening long entries and counting additional matches when the list is full. A Transformers refusal names the model path or ID. With trust off, the launcher reports the file and auto_map entry when it recognizes the declaration involved in a text tokenizer failure. Both kinds of refusal identify --trust-remote-code. An inspection error names the path and the reason inspection failed.

The vision check refuses the following canonical types in mlx-vlm 0.7.4. Their processors have paths that request or default to trusted tokenizer loading. The check uses the top-level model_type when truthy, otherwise speculators_model_type, then lowercases and applies mlx-vlm's remapping.

Canonical type Processor behavior
got, inkling (including inkling_mm_model) Requests trust for its tokenizer.
internvl_chat Requests trust when given a tokenizer name.
minicpmo, minicpmv4_6 After a built-in tokenizer failure, takes the trusted fallback only when trust is enabled.
moondream2 (including moondream1) Loads a tokenizer from another repository with trust enabled.
moondream3 Can fall back to another repository's tokenizer. Preserves explicit trust and defaults to true when the argument is omitted.

The original offline probe recorded 560 cases across MLX and oMLX, comprising 538 model or engine load attempts and 22 standalone tokenizer loads. The stand-in modules recorded import stacks and supplied minimal loader classes. The 21 drafter loads were measured separately.

Five of the 45 fetched catalog configs declare a non-empty auto_map, including mlx-community/MiniMax-M2.7-4bit and mlx-community/DeepSeek-R1-0528-4bit. Core lists both conversions as MLX variants. Their auto_map declarations pass the default mlx-lm directory check. None of the 45 fetched configs declares model_file. Config fetches failed for three catalog repositories, cognitivecomputations/Qwen3-235B-A22B-Thinking-2507-AWQ, koushd/Qwen3-235B-A22B-Instruct-2507-AWQ and QuixiAI/Kimi-K2-AWQ.

These checks assume other processes running as the same user leave checkpoint directories and files alone during startup. Some mlx-vlm processors render checkpoint chat templates outside a sandbox. That behavior was identified by source inspection and remains open under ENG-18520. The two-node launcher (kamiwaza-mlx-distributed) and kamiwaza-mlx-distributed-dflash-server retain their existing behavior. Neither distributed command was measured. oMLX uses its own trust_remote_code model setting.

Choosing the runtime: MLX or oMLX

kamiwaza-mlx-server can serve a model itself (mlx-lm / mlx-vlm in this process) or hand it to oMLX (Apache-2.0), which runs from its own environment.

kamiwaza-mlx-server -m <model dir>                  # --runtime auto (default)
kamiwaza-mlx-server -m <model dir> --runtime mlx    # this server, always
kamiwaza-mlx-server -m <model dir> --runtime omlx   # oMLX, always

auto reads model_type from the model's config.json. The shipped kamiwaza_mlx/config/runtimes.json gives the following choices.

model_type Model auto picks
deepseek_v41 DeepSeek V4.1, including Flash oMLX
qwen4_exp Qwen3.8-Flash-Next oMLX
A model that auto would otherwise serve with oMLX, launched with an option that selects MLX, such as --trust-remote-code, --ple-store, --draft-model, --dflash-draft-model or --distributed (see "Options of the MLX runtime on oMLX" for flags and environment variables) MLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice, for example runtime=mlx (auto: model_type=qwen4_exp, --ple-store given, mlx only)
qwen4_exp whose config.json declares text_config.ple_storage or llm_config.ple_storage (the mlx-vlm external-table layout, our -extple directories) MLX: oMLX cannot load that layout. --runtime omlx on it is refused
inkling2 Inkling 2 Small oMLX (refused for now, see below)
anything else, or no local config.json MLX

The table is data. To change a default, edit that file or point KAMIWAZA_MLX_RUNTIME_TABLE at your own copy. KAMIWAZA_MLX_RUNTIME sets the option from the environment. The command line wins. The runtime line says which runtime the launcher chose and why.

The launcher never falls back from one runtime to the other after a failure. If oMLX is selected and cannot be used, it exits with status 2 and prints the reason and the fix. When an option or checkpoint layout selects MLX, the runtime line names it before the model loads.

Upgrading from a build with fixed sampling defaults. On the MLX runtime, each omitted or null sampling field now defaults to its valid checkpoint value when available, even if the request sets the other fields. For example, a request sending only temperature to Qwen3.8-Flash-Next now uses top_p=0.95 and top_k=20 when those values are in the loaded checkpoint's generation_config.json. To use the old defaults, start the server with --sampling-defaults launcher or send {"temperature": 1.0, "top_p": 1.0, "top_k": 0} in the request. Earlier builds ignored the request's top_k. The launcher now applies it on backends that support top-k. For a non-null top_k, the server rejects negative integers, booleans and other non-integers. Requests with -1, true or 20.0 return HTTP 422. Some servers accept -1 for no limit. Send 0 for no top-k limit here.

Upgrading from a build without vision prompt reuse. On the MLX runtime, a qwen4_exp vision-language model can now reuse matching prompt prefixes by default when KV caching is enabled. The launcher checks reuse after the model has loaded and any native vision drafter has passed its startup self-test. Only MTP or no drafter qualifies. With --vlm-prompt-reuse on, startup requires a working checkpoint manager. A checkpoint can still exceed the memory cap. The default can keep up to two checkpoints whose cache buffers total at most 8 GiB, and each active restore needs memory for its own copy. Requests with images or media placeholder strings in the rendered prompt read the full prompt. Use --vlm-prompt-reuse off for a full read on every vision request.

On development commit 9174b27, the Qwen3.8-Flash-Next assistant messages matched with reuse on and off in one Mac Studio run at temperature 0, both without a drafter and with the MTP drafter. The comparison left out the tool-call IDs the server generates per response. The measurements are under "Qwen3.8-Flash-Next with a drafter".

Upgrading from 0.2.7 or earlier. --runtime is new in 0.2.9, the first release published after 0.2.7 (0.2.8 was a version number used during development and was not published). Up to 0.2.7 the launcher used only MLX. With --runtime auto, a local checkpoint identified as DeepSeek V4.1 or Qwen3.8-Flash-Next now goes to oMLX unless its layout or the launch options select MLX. For a hub ID, auto starts with the table default (MLX in the shipped table). A host without the oMLX environment refuses an oMLX launch (exit 2, with the install command) until the environment is installed or the launch says --runtime mlx. A launch that passes other options of this server (KV cache, tool parser, tracing and so on) for one of these models is refused on oMLX too, naming the option; add --runtime mlx to keep it on this server. Abbreviated options (--mod for --model, for example) are no longer accepted by this server on either runtime; spell options out.

The new --draft-model option enables a native mlx-vlm drafter. An explicit --draft-block-size must be at least 2 because one token comes from the target and the rest are drafted. Startup requires a completed draft round in up to three greedy text generations of up to eight tokens each. The launch logs the round count on success and exits 2 if generation raises or none of the three prompts runs a round. On oMLX, a Qwen3.8-Flash-Next launch with a truthy mtp_enabled setting requires an embedded draft-head tensor name in the checkpoint metadata. On a launch that serves on the MLX runtime, kamiwaza-mlx-server reads its bundled config/reasoning.json before it loads the model and exits 2 on a file-read error, a JSON decoding error or a failed check of the table. A launch that goes to oMLX does not read it.

Chat requests on the MLX runtime have these changes (see also "Reasoning controls").

  • With a native drafter, an unset, null, zero or negative token limit uses the served context window minus the prompt text token count, at least 1, or 2**31 - 1 if the window is unknown. Image tokens are excluded, so an image request can get more room than remains in the window. The vision path reports stop for an ordinary answer that reaches this bound. Generation keeps its previous limits for positive values or a launch without a native drafter. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens.
  • The kamiwaza-mlx-server chat endpoint on the MLX runtime now accepts max_completion_tokens as an alias for max_tokens. Version 0.2.7 ignored it. A null for either name now means unset, though max_tokens: null gave a 422 in 0.2.7. Two non-null values must match or the server returns a 400.
  • 0.2.7 ignored reasoning_effort and chat_template_kwargs. On a launch that passes them to the chat template, the server now passes the merged arguments through unless it has recorded reasoning facts for the model. A native vision-language launch adds its thinking-off default only when neither reasoning field is set. On Qwen3.8-Flash-Next, low, medium and xhigh sent alone turn reasoning on at that level. minimal is sent as low, high as xhigh, and none turns reasoning off. On the Qwen native vision-language path, enable_thinking: true sent alone now reaches the template.
  • With stripping on and auto-detection on, whether an answer starts inside a think block is now decided per request from the think tag that ends its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next with a drafter"). Visible effects: a request rendered with thinking off (as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer that could come back empty before; a vision-language stream without tools rendered with thinking on no longer sends the reasoning and </think> in content (a block of more than 50 tokens with no think tag that a token limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the prompt, is withheld on every path, including one-piece answers and streams that used to send it; and a stripped vision-language stream that sees a </think> outside a think block now updates the process-wide detection result, as the other paths already did. Text and DFlash streams now keep visible text that comes before an opening tag in the same piece, and streams without tools recognize a closing tag split across pieces inside a block. Streams without tools withhold a block whose opening tag they recognized and that does not close; text and DFlash streams used to send a long one at the end. Vision-language streams match think tags in any case and remove a </think> that has no opening tag when it arrives in one piece. Think tags are now found in the original text, so a character whose lower case is longer (for example İ) no longer shifts the split; this applies with stripping or auto-detection off too.
  • Contradictory fields, reserved template argument names and unrecognized Qwen3.8-Flash-Next efforts give a 400. Nested enable_thinking and reasoning_effort are allowed even when the renderer names them as parameters. When the fields are consistent and otherwise valid, the unrecognized-effort error lists the accepted names and mappings. When a comparison render attributes an error to the arguments, the 400 quotes the template error. In 0.2.7 a text renderer's TypeError could cause a retry without enable_thinking. Request validation can give a 422 before these checks.
  • A text renderer's TypeError, ValueError or Jinja template error that remains after tool fallback and field checks now gives a 503 quoting the error in place of an opaque 500. A 503 from the native template renderer also quotes its error.
  • A launch that uses manual text formatting or mlx-vlm's non-native prompt helper refuses reasoning fields with a 400. This includes enable_thinking, which 0.2.7 accepted there and ignored.

For an unforced minimax-m2 request on a text template that takes tools, the server renders the same request with a unique name for each tool. It uses the original prompt only if the second render succeeds, contains every marker and matches the original exactly after restoring the real names. In the renamed render each name must be the name of a JSON declaration inside its own <tool>...</tool> block. Otherwise it renders explicit instructions with the request's own arguments. The parser reads only MiniMax's call format, so a template that presents tools another way (Qwen3-Coder's, for example) gets explicit instructions. A tools-only comparison that raises TypeError also uses this fallback. MiniMax always uses explicit instructions for a forced choice, even on a template that prints <tool> blocks. Explicit instructions list every supplied tool in request order, even for a forced choice. A missing declaration or missing instructions in the rendered prompt give a 400. On these explicit-instruction paths, a forced tool_choice naming a tool absent from the request's tools is also a 400. With --tool-call-parser openai-json, a forced choice now lists every supplied tool before the forced instruction. Previously it showed only the selected function's schema. With tools enabled, openai-json now returns a 400 for an otherwise valid tool with a function member but a type other than function, with or without a forced choice. In 0.2.7, openai-json could silently omit non-function tools.

From a 0.2.8 development build to 0.2.9. These changes are relative to the last 0.2.8 development snapshot (fef2c29).

  • kamiwaza-mlx-distributed and kamiwaza-mlx-distributed-dflash-server now exit 2 when KAMIWAZA_MLX_RUNTIME is set to something that is not auto, mlx or omlx. 0.2.8 ignored such a value there.
  • A single-node launch given --distributed-world-size 1 (or given --distributed-host or --distributed-world-size while the environment turns distributed serving off) has its port checked before the model is loaded, like any other single-node launch. In 0.2.8 a taken port was refused for such a launch only after the load.
  • kamiwaza-mlx-server --help prints the option list beside a settings file that cannot be read. 0.2.8 exited 2 without it.
  • If the null device cannot be used when the server ends, the non-blocking flag is left on its output pipe (see "Tying the server to its supervisor"). In 0.2.8 the server could hang at exit in that case.
  • On a launch that serves on the MLX runtime, kamiwaza-mlx-server reads its bundled config/reasoning.json before it loads the model and exits 2 on a file-read error, a JSON decoding error or a failed check of the table. A launch that goes to oMLX does not read it.
  • With a native MLX drafter, omitting the limit no longer cuts the answer to one token. An unset, null, zero or negative limit uses the served context window minus the prompt text token count, at least 1, or 2**31 - 1 if the window is unknown. Image tokens are excluded, so an image request can get more room than remains in the window. The vision path still reports stop for an ordinary answer that reaches this bound. Generation keeps its previous limits for positive values or a launch without a native drafter. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens.
  • The kamiwaza-mlx-server chat endpoint on the MLX runtime now accepts max_completion_tokens as an alias for max_tokens. A null for either name now means unset, though max_tokens: null gave a 422 in 0.2.8. Two different non-null values now give a 400 where 0.2.8 ignored max_completion_tokens.
  • On kamiwaza-mlx-server, reasoning_effort and chat_template_kwargs now reach the model's chat template where the launch supports them. A native vision-language launch adds its thinking-off default only when neither reasoning field is set. On Qwen3.8-Flash-Next, low, medium and xhigh sent alone turn reasoning on at that level. minimal is sent as low, high as xhigh, and none turns reasoning off.
  • With stripping on and auto-detection on, whether an answer starts inside a think block is now decided per request from the think tag that ends its rendered prompt, on every path (the rule is under "Qwen3.8-Flash-Next with a drafter"). Visible effects: a request rendered with thinking off (as on the default Qwen3.8-Flash-Next MLX launch) keeps a short answer that could come back empty before; a vision-language stream without tools rendered with thinking on no longer sends the reasoning and </think> in content (a block of more than 50 tokens with no think tag that a token limit ends is still returned as the answer); a short output with no think tag, in a block inferred from the prompt, is withheld on every path, including one-piece answers and streams that used to send it; and a stripped vision-language stream that sees a </think> outside a think block now updates the process-wide detection result, as the other paths already did. Text and DFlash streams now keep visible text that comes before an opening tag in the same piece, and streams without tools recognize a closing tag split across pieces inside a block. Streams without tools withhold a block whose opening tag they recognized and that does not close; text and DFlash streams used to send a long one at the end. Vision-language streams match think tags in any case and remove a </think> that has no opening tag when it arrives in one piece. Think tags are now found in the original text, so a character whose lower case is longer (for example İ) no longer shifts the split; this applies with stripping or auto-detection off too.
  • Contradictory enable_thinking and reasoning_effort values now give a 400.
  • A reserved name in chat_template_kwargs on a template-backed launch now gives a 400. Nested enable_thinking and reasoning_effort are allowed even when the renderer names them as parameters.
  • An otherwise valid Qwen3.8-Flash-Next request on a template-backed launch with an unrecognized effort now gets a 400 listing the accepted names and mappings.
  • A launch that cannot pass enable_thinking, reasoning_effort or chat_template_kwargs to a template now gives a 400 if any is set.
  • A text renderer's TypeError could previously drop enable_thinking on retry. When the request's own render fails and a comparison render without the request's arguments succeeds, the request now gets a 400 quoting the error.
  • A text renderer's TypeError, ValueError or Jinja template error that remains after tool fallback and field checks now gives a 503 quoting the error in place of an opaque 500. A 503 from the native template renderer now quotes its error too.
  • For an unforced minimax-m2 request on a text template that takes tools, the server tries a second render with a unique name for each tool. It uses the original prompt only if that render succeeds, contains every marker and matches the original exactly after restoring the real names. In the renamed render each name must be the name of a JSON declaration inside its own <tool>...</tool> block. Otherwise it renders explicit instructions with the request's own arguments. The parser reads only MiniMax's call format, so a template that presents tools another way (Qwen3-Coder's, for example) gets explicit instructions. A tools-only comparison that raises TypeError also uses this fallback. A forced MiniMax choice now lists every supplied tool in request order before the explicit instruction to call the selected function, even on a template that prints <tool> blocks. The server refuses instructions that omit a supplied tool or disappear from the rendered prompt. On explicit-instruction paths, a forced tool_choice naming a tool absent from the request's tools now gets a 400. The 0.2.8 build could answer with tools missing. With --tool-call-parser openai-json, a forced choice now lists every supplied tool before the forced instruction. Previously it showed only the selected function's schema. With tools enabled, openai-json now returns a 400 for an otherwise valid tool with a function member but a type other than function, with or without a forced choice.
  • A --draft-model launch now exits 2 if a greedy text self-test raises or all three test generations of up to eight tokens complete without a draft round. A successful self-test logs the number of completed rounds.
  • A --draft-model launch now refuses --draft-block-size 1 with exit 2 because mlx-vlm would draft nothing and end every answer after its first token.
  • With a truthy mtp_enabled setting, an oMLX launch of Qwen3.8-Flash-Next now exits 2 if the checkpoint metadata lists no embedded draft-head tensor. The 0.2.8 launcher did not check the metadata.

The oMLX environment

oMLX is not a dependency of this package. It requires an exact mlx build that differs from this package's, and it is published as release wheels on GitHub, not on PyPI. Install it once per host, beside this environment:

python -m kamiwaza_mlx.omlx_backend install --prefix <root>/omlx-venv   # needs uv
python -m kamiwaza_mlx.omlx_backend check

install downloads the pinned oMLX release wheel, verifies its SHA-256, and then installs it with uv together with the locked dependency set shipped in this package: 108 packages from PyPI at exact versions and four built from pinned git commits. Only the oMLX wheel is checked against a hash; the other packages are pinned by version or commit, not by hash. It needs network access and refuses to touch an existing environment. An installation that fails part-way (a network error while the dependencies are fetched, for example) can be finished by running the same install command again: the installer marks the directory when it begins and when it has finished, and a directory it began and did not finish is installed into again. Until then check and every launch refuse that environment as incomplete, by name. Nothing is deleted by the installer; a directory it did not begin is left as it is. The pin (version, wheel URL, hash, lock file) is in kamiwaza_mlx/config/omlx.json; today it is oMLX 0.7.3 with mlx==0.32.4 on Python 3.12. The launcher looks for the environment in KAMIWAZA_OMLX_HOME, or omlx-venv next to the environment this package is installed in, and refuses any other oMLX version.

Inkling 2 Small (inkling2) needs oMLX support that is not in any published release yet (upstream commit adc3daa6, PR #5229). Until the pin moves to a release that has it, the launcher exits and says so. The regular MLX runtime cannot serve it either: mlx-lm and mlx-vlm at this package's pins have no inkling2 model type.

What a launch on oMLX does

  • Everything oMLX reads or writes is under one state directory per model: KAMIWAZA_MLX_OMLX_STATE_DIR/<model directory name>-<key> (default ~/.kamiwaza-mlx/omlx/...), where the key is derived from the model directory's full path, so two models whose directories share a name do not share state, and the same model finds its cache again. One server at a time uses a state directory: a second launch of the same model directory while the first is running exits 2; give it its own KAMIWAZA_MLX_OMLX_STATE_DIR to run two copies. It holds the oMLX profile, a view of the model made of links (no weights are copied), and the prompt-prefix cache. Nothing is written to ~/.omlx, and OMLX_* environment variables are not passed on.
  • oMLX's start-up clean-up of distributed rank processes is switched off. Left on, it terminates ranks recorded under ~/.omlx whose coordinator has exited, and those can belong to another oMLX installation on the same Mac.
  • oMLX's own /health answers without a key and includes the model's name and memory figures; the MLX runtime's /health returns the status only.
  • Only inference routes are served (/health, /v1/models, /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/messages, /v1/messages/count_tokens, /v1/responses and /v1/responses/<id>, /v1/tokenize, /v1/detokenize). oMLX's administration pages and API answer 404, and its WebSocket routes (realtime audio) are closed before they are accepted.
  • For DeepSeek V4.1 a request is held to the model's native context: prompt tokens plus the output budget plus a small reserve may not exceed 1,048,576, or the request is answered 400 before anything is generated. The answer is oMLX's own overflow error (code context_length_exceeded, message opening "Prompt is too long: N tokens > 1048576 maximum context length"), so clients that compact a conversation on overflow do so; it names the field the route takes its budget from (on chat both spellings, max_tokens and max_completion_tokens, since oMLX's parsed request does not record which one was sent). The check uses what oMLX itself resolved: the request oMLX parsed and validated and the budget field its route generates with, at the point where each generating route (/v1/completions, /v1/chat/completions, /v1/messages, /v1/responses, streaming or not) checks the prompt against the context window. The launcher does not parse request bodies. At start-up it verifies that the installed oMLX still has that point and exits 3 naming what differs if it does not.
  • No request body over KAMIWAZA_MLX_MAX_REQUEST_BODY_MB (default 32 MiB) reaches oMLX, on any model, whatever its content type and whether it has a Content-Length or is chunked: it is answered 413. The bytes are counted as they arrive and never buffered by the launcher. This runs before the key check, because the web framework reads a body before it checks the key.
  • Browser clients (pages that call the server from JavaScript): on oMLX the browser origins http://127.0.0.1:<port> and http://localhost:<port> are allowed and no other, whatever --host is; there is no setting for it. The MLX runtime sends no cross-origin headers at all, so a browser blocks every cross-origin read from it. A page served from another origin therefore works against neither runtime directly. Non-browser clients are not affected.
  • Requests need no API key unless one is configured (see "API key" below).
  • The model is loaded at start-up. /health answers 503 ("status": "loading") until it is loaded and 200 afterwards. If the load fails, the process prints the reason and exits with status 4; it never stays up without a model. While the model is loading, an inference request gets 503 at once, with Retry-After: 5 and {"error": {"code": "model_loading", ...}}; it is not held until the load ends. This answer, like the 404 for routes that are not served, comes before any API-key check. Both are given by a plain ASGI wrapper that looks at the method and the path only and does not wrap a request it lets through (the body-size count above is the one thing the launcher puts around a request), so a streamed answer and a client that disconnects are oMLX's own to handle. A stop signal (SIGTERM, SIGINT) during the load ends the process within a second or two (status 143 or 130) without finishing the load.
  • The model is served under its directory name, one request at a time; later requests wait.
  • The shipped oMLX profile turns reasoning off for Qwen3.8-Flash-Next. Send "enable_thinking": true (or the same key in chat_template_kwargs) to turn it on. On the default MLX launch, a recognized effort level or alias alone also turns it on. DeepSeek V4.1 reasons unless the request turns it off, as in the deployment it was validated with. See "Reasoning controls" for the levels and the differences between runtimes.
  • Drafting on oMLX uses a multi-token-prediction head stored inside the checkpoint, turned on with KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. For Qwen3.8-Flash-Next, a truthy mtp_enabled setting requires a tensor name starting with mtp., language_model.mtp., model.mtp. or model.language_model.mtp.. The launcher checks the safetensors index if it can read it as a JSON object, or the shard headers otherwise. If it finds no matching name, it exits 2. To serve without drafting, set mtp_enabled to false. To use a separate drafter directory, see "Qwen3.8-Flash-Next with a drafter".
  • /v1/models reports the context oMLX serves, which is the model's own (262,144 for Qwen3.8-Flash-Next). The MLX runtime reports 131,072 for the same directory because that is the window it serves by default (--kv-cache-max-tokens, at most 128K unless raised). Each number is true for its runtime.
  • The launcher process becomes the oMLX process (same PID).
Variable Meaning
KAMIWAZA_OMLX_HOME The oMLX environment
KAMIWAZA_MLX_OMLX_STATE_DIR Parent of the per-model state directories
KAMIWAZA_MLX_OMLX_SSD_CACHE_MAX Size of the on-disk prefix cache (default 20GB)
KAMIWAZA_MLX_OMLX_MEMORY_CEILING_GB oMLX memory-guard ceiling (DeepSeek V4.1 defaults to 400, for a 512 GB Mac)
KAMIWAZA_MLX_MAX_REQUEST_BODY_MB The largest request body passed on to oMLX, in MiB (default 32; 1 to 512). A larger body is answered 413. Text requests are far below the default; a vision model that is sent images or video as base64 may need more. 512 is oMLX's own limit, which the launcher does not change; a larger value is refused at start (exit 2)
KAMIWAZA_MLX_OMLX_MODEL_SETTINGS JSON object merged into the oMLX model settings, e.g. {"mtp_enabled": true}. is_pinned cannot be turned off. A truthy mtp_enabled on Qwen3.8-Flash-Next requires a draft-head tensor name in the checkpoint metadata (exit 2 if absent)

API key

KAMIWAZA_MLX_API_KEY makes the server require a key, on either runtime: every route except GET /health then needs Authorization: Bearer <key> (or x-api-key: <key>), and answers 401 without it. CORS preflight (OPTIONS) requests carry no key and are let through. With no key set, the server is open, as before. The earlier name KAMIWAZA_MLX_OMLX_API_KEY is still accepted; if both are set to different values the launcher exits 2. The key can also be written in a .env file in the directory the server is started from; the process environment wins over the file. That file is read the same way on both runtimes and by the launchers that refuse a key. oMLX's own OMLX_API_KEY is not used: every entry point, on either runtime, exits 2 while it is set (in the environment, in the settings file, or passed with --env) and names the variable to use. The rule behind it: no entry point starts listening without authentication while a key variable it knows about is set. The start-up output says whether a key is in force: on the MLX runtime API key authentication: enabled (every route except /health). or API key authentication: off.; on oMLX kamiwaza-mlx: inference API key: required or kamiwaza-mlx: inference API key: off (requests are not authenticated). (oMLX's own API key authentication: enabled line, which it would print on every launch because the launcher always sets its internal management key, is reworded to oMLX management key: set (internal; ...).)

A program that imports kamiwaza_mlx.server to serve inside its own process is held to the same rule, and it fails closed: the key is decided from the process environment and the settings file together, so a key in either is enforced, and where the configuration cannot be trusted (a settings file that cannot be read, OMLX_API_KEY set, two key variables that disagree, a key in a ./.env that a named file replaces) the import ends with SystemExit(2) and no server starts. Such a program cannot turn the key off by leaving the settings file unloaded.

The key is never logged and is not written to the launch description; on oMLX it is in the profile's settings.json (mode 600). The key applies to kamiwaza-mlx-server. The two-node launcher (kamiwaza-mlx-distributed) and kamiwaza-mlx-distributed-dflash-server serve through mlx_lm.server, which checks no key, so they refuse to start while the variable is set.

GET /health exists on both runtimes. On oMLX it answers 503 while the model is loading and 200 once it is loaded. The MLX runtime does not listen until its load step has returned, so there 200 means the server is up; a large model loaded lazily still reads its weights on the first request, which is slow.

Tying the server to its supervisor

By default the server keeps running if the process that started it dies. A supervisor that must never leave a model server behind passes one of:

Option The server stops when
--exit-with-parent the process that started the launcher exits
--parent-pid <pid> the process with that PID exits (for a supervisor that is not the direct parent, for example behind a wrapper or nohup)

The contract, on both runtimes:

  • The watched PID is fixed when the launcher starts and is never re-read. The launcher prints kamiwaza-mlx: this server stops when process <pid> exits.

  • If that process does not exist at start, or --exit-with-parent is used when the parent is already PID 1, the launcher exits 2 and starts nothing.

  • When the watched process exits, however it exits, the server prints kamiwaza-mlx: the supervising process <pid> is gone; stopping this server. (on oMLX always; on the MLX runtime while the web server runs, and before that the process ends without the line), sends itself SIGTERM, and if it is still running 5 seconds later exits with status 5. Those 5 seconds are fixed: KAMIWAZA_MLX_STOP_GRACE_SECONDS does not lengthen them, so with a supervisor gone a request in flight gets at most 5 seconds. During a model load it stops at once.

  • The watch covers the model load and, on oMLX, lives in the process that remains after the launcher replaces itself (same PID as the launcher).

  • It watches for the process exiting, not for a signal sent to it: a supervisor that is stopped (SIGSTOP) or hung still counts as alive.

  • Output is best effort when the pipe it goes to is full, and no write waits on it for longer than a set bound (below). A supervisor usually reads the server's stdout and stderr through pipes; once it is gone a write to them fails, and if it stops reading, a blocking write waits forever once the pipe is full. So, when (and only when) descriptor 1 or 2 is a pipe or a socket, two processes make that descriptor non-blocking and replace sys.stdout / sys.stderr with a thin wrapper around it:

    Process Best-effort output
    kamiwaza-mlx-server on the MLX runtime (console command or python -m kamiwaza_mlx.server) yes
    The oMLX process the launcher becomes (--runtime omlx) yes
    kamiwaza-mlx-distributed (it starts the ranks and the proxy on the same pipe and is replaced by mlx.launch) no: ordinary blocking output
    kamiwaza-mlx-distributed-dflash-server (one rank among others on the same pipe) no: ordinary blocking output
    A program that imports kamiwaza_mlx.server no: the import replaces no stream and installs no error hook and no signal handler, and its refusals are not hard exits (see "Importing the server module" below for what it does do)

    In those two processes a write goes straight to the descriptor. If the pipe is full the write waits for it for at most 50 ms in total, then drops the rest. For the next second no write waits: each puts what fits into the pipe and drops the rest. So once a write has waited its full 50 ms in vain, no thread waits again for a second. That does not bound a reader that is slow but gets there: a write that the reader takes after, say, 30 ms waits those 30 ms and loses nothing, and so does the next one, so such a reader slows the server down by that much per write. No write raises, and flush does nothing. Once that second has passed, a write adds one line saying what was lost: [kamiwaza-mlx: output could not be delivered on <stderr>: N writes (M bytes) were dropped], at most once every 5 seconds; loss right before the process exits is not reported. A line can be cut short where the pipe filled up, and so can that line. The bound for a reader is those 50 ms per write: output is complete when the reader takes every write within 50 ms (tested with 5,000 log lines; an 8 MiB burst of 1 KiB lines printed as fast as possible to a reader that reads as fast as it can was delivered whole when measured on the development machine, macOS on Apple silicon; the figure depends on the machine's pipe size and scheduling). A reader that pauses longer or starts late loses what did not fit in the pipe meanwhile, and a single write too large to be taken within 50 ms loses its tail. A terminal and a regular file are left exactly as they are, descriptor and stream.

  • Side effects of the non-blocking flag, which belongs to the pipe end and not to the process. While the server runs, anything else that writes through the same pipe end sees it: native code in the server process gets "would block" where it used to wait (intended), and so does a child process or a sibling in the same shell pipeline that shares the server's stdout or stderr. The reading end is not affected. Give the server its own pipe if something else must write to it with blocking semantics while the server runs.

  • The flag is set late and taken back. kamiwaza-mlx-server sets it only once the runtime choice has decided that this process serves on the MLX runtime, and the oMLX process sets it when it starts: a launch refused by the runtime choice (a bad option, a setting, a key, a taken port, a missing oMLX environment) leaves the pipe as it was. The flag is cleared again, where this process set it, when the process ends by a normal exit, a refusal or any other exit with a status of its own (load failed, stop deadline, supervisor gone, start-up error), and by SIGTERM after the web server's own stop. Just before, the process points its descriptors 1 and 2 at the null device, so from that moment nothing it writes reaches the pipe (a write to a blocking pipe that is full and unread would wait for good, and the interpreter still writes while it shuts down). The null device is opened when the process replaces its output streams, not when the process ends. If descriptor 1 or 2 cannot be pointed at it (the null device could not be opened, or the process has no file descriptor left), no flag is cleared: a process that cannot hang matters more than the flag. It is also not cleared where the process gets no chance to: after SIGKILL or another signal that ends a process by default (SIGHUP, SIGQUIT and the like), after a crash in native code or code that calls os._exit on its own, and after a SIGTERM during the MLX runtime's start-up (from the runtime choice until the server serves, the model load included), which ends the process at once, by design. A program that writes to the same pipe after one of those finds it non-blocking; a supervisor that runs further commands on a server's pipe can clear the flag itself (os.set_blocking(fd, True)).

  • Several servers on one pipe. Each sets the flag if it is not set and clears it on leaving only if it set it; every write first sets it again if another server has cleared it meanwhile. So after one of them leaves, the pipe is blocking until the next write of one that still runs, and it is blocking after the last has left, in whatever order they leave. One window remains: a server that leaves at the very moment another is between that check and its write (the check is made before a write and again after a wait), with the pipe full and unread, lets that one write wait for the reader.

  • Importing the server module. For a program that imports kamiwaza_mlx.server, the module body runs startup and main_entry() serves. Argparse handles usage errors. Startup refusals, including native drafter load and self-test failures, raise a catchable SystemExit. Other target-load errors and interrupts propagate. Once installed with trust off, the Transformers guard stays active in that process. The settings file (the one named with --distributed-env-file, else ./.env) is loaded where it always was (when distributed settings are resolved, after the options have been parsed), into the environment for names not already set; it is not loaded early for every setting as the console command does, so an option default taken from the environment does not see it. Differences from before: the file is read by the parser described under "Every setting on every path" (a line that is not a setting is an error instead of being skipped; only matching quotes are removed; a leading export and a byte-order mark are accepted); the refusals for a named file that is missing or empty and for the option given twice apply, at the point where the key is decided, which for an importer is after the model has loaded; and the key is decided as on the console command, from the environment and the settings file together (the file is read in place for this, not loaded), so KAMIWAZA_MLX_API_KEY in either is enforced, an empty variable in the environment does not cancel a key in the file, and the refusals for OMLX_API_KEY and for a key in a ./.env that a named file replaces apply. With --exit-with-parent or --parent-pid on its command line the importing process is watched and ended like the console command's. main_entry() itself reads no settings file.

  • On the two distributed commands a full, unread pipe can still block ordinary output (logging, the ranks' own lines). Their refusals and start-up failures are armed from another thread before their line is written, so those exits do not depend on the pipe.

  • Without either option nothing is watched; running the server from a terminal is unaffected.

Exit status

Status Meaning
1 An error at start-up that no refusal covers: one line start-up failed: ..., then the traceback. On kamiwaza-mlx-distributed also a required setting that is missing or a local preparation step that reported its failure (one line; a step that failed with an error nobody handled prints the traceback too). On oMLX also a port taken between the launcher's check and oMLX's own bind: oMLX's error, before the model is loaded
2 Refused: bad option or option value, unreadable config.json, unusable runtime, missing, incomplete or mismatched oMLX environment, unsupported model or layout, conflicting keys, a bad setting, a settings file that cannot be read, a port that cannot be bound (for a wildcard host such as 0.0.0.0, also a port another process holds on the loopback address alone; checked before the load for every launch on oMLX and every single-node launch on the MLX runtime, --distributed-world-size 1 included, and not for a distributed one, where a rank that does not serve is given the same port), supervisor already gone. Nothing is served. On the MLX runtime with trust off, the launcher checks the drafter before the target loads. It checks each target path before its loader runs, including a vision fallback after a text attempt. Transformers can refuse during loading. A drafter that cannot be resolved, inspected or loaded also exits 2. Requesting --ple-store on a text route exits 2 before weight loading. A table supplied through --ple-store that mlx-vlm rejects as incompatible exits 2. If checkpoint inspection refuses the checkpoint's own PLE manifest, startup exits 2; if inspection passes and mlx-vlm rejects that manifest during loading, it is a model-load error and exits 4. A speculative drafter that loads is also refused if a test generation raises or none of three test generations completes a draft round (see "Native speculative decoding"). Drafter initialization and test-generation failures release the loaded target model. The eligibility and manager checks for --vlm-prompt-reuse on can also refuse the launch after the model and any vision drafter have loaded and the drafter's self-test has passed. On the MLX runtime, a port taken between the check and the web server's own bind is the same refusal, after the load (on oMLX see status 1)
3 oMLX start-up could not be made safe: a start-up guard could not be applied, the profile could not be written, or the supervising process exited in the moment between the launcher's check and oMLX starting
4 The model could not be loaded, including an ordinary vision-model load failure. MLX summarizes the error and up to four of its causes on one line and logs the traceback at debug level. oMLX reports when it finds no pinned model to load. If loading a pinned model raises an exception, it reports the exception message or, if empty, its type.
5 The supervising process exited and the server did not stop within 5 seconds of its own SIGTERM
143 / 130, or ended by the signal Stopped by SIGTERM / SIGINT. See the next table: the same stop reaches a supervisor in more than one form

What a supervisor sees after a stop

The rule: a stop that the web server completes by itself ends the way the web server ends it, and a stop that the launcher has to complete ends with an exit status. Concretely:

  • While serving, the first SIGTERM or SIGINT starts the stop deadline, KAMIWAZA_MLX_STOP_GRACE_SECONDS (default 10; 0 to 3600), on either runtime. If every request finishes in that time the web server stops by itself; if not, the process exits when the time is up with 128 plus the signal (143, 130), also when its output can no longer be written. KAMIWAZA_MLX_OMLX_STOP_GRACE_SECONDS is accepted as another name for the setting; the two set to different values are refused.
  • During a model load a stop is immediate.
  • When the supervising process is gone (--exit-with-parent, --parent-pid), the server sends itself SIGTERM, which is a stop like any other and so starts the stop deadline, and it exits with status 5 if it is still there 5 seconds later. Whichever comes first ends it: the web server's own stop, the stop deadline (when the grace is under 5 seconds), or status 5.

A supervisor that reads the wait status (Python's Popen.returncode, for example) therefore sees an orderly stop and a stop by the deadline differently, where a shell shows 143 for both. Treat "killed by SIGTERM" and exit status 143 as the same outcome: the server stopped because it was asked to. The cases that have been observed:

Case Wait status Observed
SIGTERM while serving, orderly stop, either runtime killed by SIGTERM (returncode -15) live, both runtimes
SIGTERM while serving, a request still in flight when the grace ends exit status 143 live on oMLX; on the MLX runtime by test with a real web server
SIGINT while serving on oMLX, orderly or by the deadline exit status 130 live
SIGINT while serving on the MLX runtime, orderly stop exit status 0 live
SIGINT while serving on the MLX runtime, a request still in flight when the grace ends exit status 130 by test with a real web server
SIGTERM / SIGINT while oMLX is loading the model exit status 143 / 130 live
The model could not be loaded exit status 4 live
Refused at start exit status 2 (3 for an oMLX start-up guard) live
The supervising process was killed gone within 0.1 to 6.1 seconds, both runtimes live; the status has no observer left

Not observed, from the code: SIGTERM while the MLX runtime is loading the model ends the process killed by SIGTERM; SIGINT at any point before the MLX runtime serves (during the load, or between the load and serving) ends it with exit status 130.

Every setting on every path

docs/settings-parity.md lists the settings the package reads (a test finds them by scanning the source for environment variables read through its helpers and for the options of its parsers, and fails when one is in no row). It sets them (the API key, distributed settings, listener, runtime choice, the MLX-only options, oMLX settings, oMLX's own OMLX_* variables and the rest) against every way it can arrive (command line, process environment, ./.env, --env passed to ranks, the selection table or an oMLX profile override) for each of the four entry points. Every cell is honored, refused by name, or not applicable with the reason; a variable that does not apply to an entry point is named in a start-up line (set but not used by ...). The table is generated from kamiwaza_mlx/settings_parity.py, and a test fails when a setting is added without a decision.

./.env in the directory the server is started from is loaded into the environment before any setting is read, on both runtimes; what is already in the process environment wins (an empty KAMIWAZA_MLX_API_KEY in the environment does not cancel a key in the file). A file named with --distributed-env-file is read in place of ./.env, as before; nothing is merged from the two, and a start-up line says which file is in force when both exist.

A settings file is read strictly, by one parser for all four entry points and the distributed runtime. A line is NAME=value, with optional surrounding whitespace, an optional leading export, and optional matching quotes around the value; blank lines and lines that start with # are skipped. Nothing after the = is treated as a comment: in KAMIWAZA_MLX_API_KEY=secret # prod the key is secret # prod. Because that is legal and seldom meant, a value that contains # is pointed out at start-up by file, line and variable name (never by value). A value that begins with # (NAME= # comment) would be nothing but the comment: for a key variable that is refused, naming file, line and variable; for any other it is pointed out the same way.

Where the launcher cannot be sure what was configured, it exits 2 naming the file and does not start: a settings file that exists and cannot be read or decoded; a line that is not a setting (the message gives the line number and never the line); a file named with --distributed-env-file that does not exist or holds no settings, also when it names ./.env (it does not fall back to ./.env); --distributed-env-file given more than once; and a ./.env that is not read because another file was named, while it holds an API key that the named file and the environment do not.

kamiwaza-mlx-server --help (or -h) starts nothing, so it is not refused for the settings file (./.env, or the one named with --distributed-env-file): it prints the option list, preceded by a line that says a launch with these settings would be refused and why.

kamiwaza-mlx-distributed and the DFlash distributed server do not load ./.env; they refuse a key or KAMIWAZA_MLX_RUNTIME=omlx written there, and name any other setting found only there in a start-up line. They also refuse a KAMIWAZA_MLX_RUNTIME that is not auto, mlx or omlx (a typo), in the environment or in ./.env, as kamiwaza-mlx-server does: both always serve with MLX, but a misspelled value means the configuration is not what its author meant. kamiwaza-mlx-distributed refuses it in an --env assignment too. (Up to 0.2.7 the MLX server read that file only for its distributed settings and, as a side effect, for anything it happened to read later.)

Options of the MLX runtime on oMLX

No option is dropped. Each one is either already what oMLX does, or makes auto choose the MLX runtime, or stops the launch naming the option and the runtime.

Option On oMLX
-m, --host, --port, --exit-with-parent, --parent-pid Used
--vision Accepted: oMLX detects a vision checkpoint itself
--strip-thinking Accepted: oMLX returns reasoning in a separate field
--trust-remote-code With the shipped table, the flag selects MLX when auto would otherwise choose oMLX. The runtime line names the first matching MLX-only option or the environment variable that decided the choice. Explicit --runtime omlx refuses the flag. To allow checkpoint code on oMLX, remove the flag and set KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"trust_remote_code": true}'.
--ple-store With --runtime auto, selects the MLX runtime (see the table above). With an explicit --runtime omlx: refused. oMLX keeps the n-gram table of Qwen3.8-Flash-Next on disk by itself (the launcher sets qwen4_ple_ssd_offload), but always the table in the model's own checkpoint. Pairing weights with the table of another conversion needs --runtime mlx
--draft-model, --draft-kind, --draft-block-size, --draft-queue-timeout, --draft-stream-send-timeout, --dflash-draft-model With --runtime auto, select the MLX runtime. With an explicit --runtime omlx: refused. oMLX drafts with the head embedded in the checkpoint: on by default for DeepSeek V4.1, and for other models through KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. A separate drafter directory needs the MLX runtime
--vlm-prompt-reuse, --vlm-prompt-reuse-checkpoints, --vlm-prompt-reuse-max-gb With the packaged runtime table, these settings do not select a runtime. If auto chooses oMLX or you select oMLX explicitly, supplying any of these flags or a nonblank value for its environment variable stops the launch, even for --vlm-prompt-reuse off. oMLX manages its own prompt cache. Use --runtime mlx to apply these options. There, --vlm-prompt-reuse on also requires a model type on the checked list.
--sampling-defaults oMLX manages its own sampling defaults. With the packaged runtime table, this option does not influence runtime selection. If auto or an explicit choice selects oMLX, supplying the flag or a nonblank value for its environment variable stops the launch, even for model.
--text-only Refused
--no-distributed Accepted: oMLX serves on one node
a word that is neither an option nor an option's value (... 8080) Refused, naming it
--distributed and the other --distributed-* options Under --runtime auto they select the MLX runtime, so the launch does not reach oMLX. With --runtime omlx they are refused by name
distributed MLX turned on through the environment or a .env file in the working directory (KAMI_DIST_ENABLED, MLX_DISTRIBUTED_ENABLED, PAIRED_HOST, MLX_DISTRIBUTED_HOST, MLX_DIST_HOST, or PAIRED_WORLD_SIZE / MLX_DISTRIBUTED_WORLD_SIZE above 1) Under --runtime auto it selects the MLX runtime, named in the runtime line. With --runtime omlx it is refused, naming the variable: oMLX would start a separate single-node server on every rank. A distributed variable that does not turn the mode on (RANK, PAIRED_PORT, MLX_DISTRIBUTED_TIMEOUT and the like) has no effect on oMLX and is named in the set but not used by ... line
every other option (KV cache, tool parser, tracing, ...) Refused: it configures this server, not oMLX

Sampling defaults

For stock text and vision-language generation on the MLX runtime, each of temperature, top_p and top_k is chosen independently. A non-null request value wins. For an omitted or null field, the server uses the checkpoint's value, then falls back to temperature=1.0, top_p=1.0 or top_k=0 (no top-k limit). An explicit zero counts as a value, as does 1.0 for temperature or top_p. In a request trace, request normally holds the validated request with schema defaults before sampling defaults are applied. Omitted sampling fields are null. If the server cannot serialize the request, this field holds only model. resolved_sampling holds the resolved values of temperature, top_p and top_k.

Flag Environment variable Default Meaning
--sampling-defaults {model,launcher} KAMIWAZA_MLX_SAMPLING_DEFAULTS model Use the checkpoint's sampling defaults. Select launcher to use the fixed defaults for omitted or null fields.

In model mode the launcher reads generation_config.json from the loaded vision model's model_path if it names an existing directory. Otherwise it uses the launcher's resolved model directory. Its last choice is the loaded tokenizer's name_or_path if that names an existing directory. If no directory is known, the startup log reports model directory unknown and the launcher uses its fixed defaults.

For checkpoint values, temperature must be finite and nonnegative, top_p must be greater than zero and at most one, and top_k must be a nonnegative integer. An invalid field produces a warning and uses its launcher default. The startup log lists each default and its source. The launcher ignores do_sample and other keys when choosing these defaults. Both pinned model loaders also open the file. An unreadable file can prevent startup, and mlx-lm can also reject JSON that is not an object.

For request values, a supplied, non-null top_k must be a nonnegative integer. The request accepts numeric temperature and top_p outside the checkpoint ranges. The pinned mlx-lm and autoregressive mlx-vlm samplers apply top-p filtering only for 0 < top_p < 1. Zero temperature selects greedy decoding. With a negative temperature the sampler divides the scores by that value, reversing their order. If a top-k or top-p filter is active, including one taken from the checkpoint, the tokens it filtered out then score highest. Autoregressive mlx-vlm, including Flash-Next, rounds positive temperatures below 0.01 up to 0.01.

For both requests and checkpoints, top_k=0 disables the cutoff. The launcher collects vocabulary sizes from model configuration, output weights and tokenizer length. A cutoff at or above the largest known size also means no limit, with a startup warning for a checkpoint value. With no known size, the launcher passes the cutoff through. When an autoregressive sampler applies top-k filtering, a positive cutoff at or above the actual output width raises an error. With an EAGLE-3 drafter whose vocabulary is smaller than the target's, mlx-vlm applies the same sampler to the draft logits, so an active cutoff at or above the draft vocabulary size raises the same error. At zero temperature the sampler returns the top token without applying the filter.

The dflash-mlx text backend discards both the temperature and the sampler, so none of temperature, top_p or top_k applies there, whether supplied by the request or the checkpoint. The --dflash-backend auto setting can also select this backend when installed. DiffusionGemma in the pinned mlx-vlm ignores top_p and top_k from both requests and checkpoints.

Tool-call parsing

With tools enabled on MLX, qwen-xml first applies its existing whole-answer grammar to the visible text. An answer consisting of valid calls and whitespace returns those calls in order with message.content set to null.

In mixed text, the parser takes every complete <tool_call>…</tool_call> block outside reasoning, including blocks in prose or Markdown fences. It skips paired <think> spans left in the visible text and everything from an unmatched opener to the end. It skips the whole mixed answer as unfinished reasoning if the generation prompt ends in <think> after trimming trailing whitespace and the raw output has no </think>. Both tag checks ignore ASCII case.

Each block uses the call-only grammar to check its function name and arguments. If a block fails to parse, or any individual gap outside reasoning contains a wrapper tag matching </?tool_call(?![A-Za-z0-9_-]), the whole set is discarded. With no calls, the server keeps the visible text unchanged and runs its existing validation on it. Tool choice and parallel-call limits apply to extracted calls. A tool-output error produces HTTP 502 in JSON or an invalid_model_tool_output event followed by [DONE] in an HTTP 200 stream.

Read both message.tool_calls and message.content. Removing the blocks leaves the original text and retained reasoning in order, with outer whitespace trimmed and an empty result represented by null. A buffered stream sends reasoning, then any remaining content in one delta, then the calls in order and the tool_calls finish reason. Argument JSON uses ASCII escapes for a call if its arguments cannot otherwise be encoded as UTF-8.

Differences from other servers

In vLLM v0.9.2, the non-streaming Hermes parser returns the text before the first call and drops text between and after calls. It accepts an unclosed final block if the text through the end decodes to a valid Hermes call object. Invalid JSON in any block yields the whole text with no calls. Excluding reasoning from Hermes input requires a configured upstream reasoning parser.

Reasoning controls

How much a model reasons is decided by its chat template. A chat request sets the template's arguments with three fields:

Field Meaning
enable_thinking true or false: reasoning on or off for this request
reasoning_effort how much to reason. The Qwen3.8-Flash-Next template uses low, medium and xhigh (its default when thinking is on, and it can think for minutes). The MLX runtime also accepts the names listed below
chat_template_kwargs an object of further template arguments, for example {"enable_thinking": true, "reasoning_effort": "low"} or {"preserve_thinking": false}. On the MLX runtime non-null values of the two fields above win over the same names inside it. A null for any of the three fields, or for enable_thinking / reasoning_effort inside the object, means "not set"

For Qwen3.8-Flash-Next, reasoning is off on the default MLX launch and in the shipped oMLX profile. On MLX an accepted effort name other than none, sent alone, turns it on. On oMLX, send "enable_thinking": true (see "On oMLX" below). The MLX runtime loads this model through mlx-vlm. The pinned mlx-lm text loader does not take this model type.

On the MLX runtime

These reasoning, token-limit and max_completion_tokens rules are those of kamiwaza-mlx-server's own chat endpoint on the MLX runtime. kamiwaza-mlx-distributed and kamiwaza-mlx-distributed-dflash-server serve through mlx-lm's own server, which handles these fields its own way.

The API validates the request before the server resolves its tool parser and reasoning fields. Invalid field types can give a 422, and an incompatible explicit qwen-xml parser gives a 503. The server checks template argument names before checking the following combinations.

  1. enable_thinking true with reasoning_effort none gives a 400.
  2. enable_thinking false with any effort other than none gives a 400.
  3. enable_thinking true and no reasoning_effort: an effort is added only for a model recorded as needing one. Qwen3.8-Flash-Next uses its template default, xhigh.
  4. An effort level and no enable_thinking: the server turns thinking on only for a model recorded as needing it for the level to take effect. Qwen3.8-Flash-Next reads the effort only while thinking is on.

Rules 1 and 2 hold for every model. What else the launcher does depends on what it knows about the model:

  • A model type with recorded reasoning facts. They are in kamiwaza_mlx/config/reasoning.json, by the model_type in the model's config.json (for a model named by a hub id, the loaded checkpoint's), and exist only where a model was measured to need them. The server accepts the model's own levels and all six standard OpenAI API effort names (none, minimal, low, medium, high and xhigh). For Qwen3.8-Flash-Next (qwen4_exp), it accepts none (enable_thinking false and no level), minimal (sent as low), low, medium, high (sent as xhigh) and xhigh. After the switch and contradiction checks, an unrecognized effort gives a 400 listing the accepted names and mappings. A level or mapped name sent alone gets enable_thinking true added (rule 4). Thinking on alone gets nothing added (rule 3).
  • Any other model type (no recorded facts, or a model whose type cannot be read). The launcher passes the parsed, merged fields through without model-specific additions, renames or effort validation. A number is passed through as an effort too. If either reasoning field is set, the server adds no default of its own. In that case the server passes the request's merged template arguments and adds no reasoning default. On a text launch that renders through mlx-lm's tokenizer wrapper, the wrapper supplies a missing enable_thinking: true if it recognizes thinking tokens in the tokenizer, false otherwise. A comparison render can identify a failure due to the request's arguments and return a 400 with the template error (see "Details"). The server logs pass-through once per process.
  • A launch where the fields would not reach a template. A request that sets any of the three is a 400. For otherwise valid fields, the error names the field and the launch. This applies to a text launch where the server formats messages manually, and to a vision-language model on a tool parser other than qwen-xml (another model, or --tool-call-parser openai-json, minimax-m2 or none), whose prompt is built by mlx-vlm's prompt helper, which this server calls without template arguments. (mlx-vlm 0.7.4's helper could be given them; doing so is a change of its own and is not made in this release.) A request for no reasoning is refused there like any other.
The request sends Recorded facts (Qwen3.8-Flash-Next) No recorded facts No template gets the fields
none of the three thinking off on the default vision-language launch enable_thinking false is sent on the native vision-language path (a template that does not read it ignores it); the loader and template default on text the launch's default
enable_thinking true reasoning on at the template's default level; nothing added to the template as sent 400
enable_thinking false reasoning off to the template as sent 400
reasoning_effort low, medium or xhigh, alone reasoning on at that level (enable_thinking true added) the server adds no default. Only the effort reaches the native vision-language template. On text, mlx-lm also sends enable_thinking, true for a tokenizer with thinking tokens and false otherwise 400
the same with enable_thinking true reasoning on at that level to the template as sent 400
reasoning_effort minimal or high, alone or with enable_thinking true reasoning on at low or xhigh, respectively the effort passes through. On text, mlx-lm supplies a missing enable_thinking 400
reasoning_effort none, alone or with enable_thinking false reasoning off the launcher passes the fields through. The template can accept, ignore or refuse them. To ask for thinking off, send enable_thinking false alone 400
any other effort (off, turbo, a number), alone or with enable_thinking true 400 listing the accepted names and mappings the effort passes through. On text, mlx-lm supplies a missing enable_thinking 400
enable_thinking true with none, or enable_thinking false with any effort other than none 400 that names the contradiction the same 400 the same 400
chat_template_kwargs permitted variables go to the template. Nested reasoning controls follow the same rules as the fields permitted variables go to the template as sent. Unrelated variables alone keep the native vision-language default 400 if any variable is set. An empty object or one containing only null reasoning controls sets nothing. A null ordinary variable is set

On the native vision-language path, the server adds its thinking-off launch default (enable_thinking false) only when the request sets neither enable_thinking nor reasoning_effort, at the top level or inside chat_template_kwargs. A recorded effort mapping can also send enable_thinking false (none on Qwen3.8-Flash-Next). Unrelated template arguments keep that default. For a text model, mlx-lm 0.31.3 supplies its loader default, true for a tokenizer with thinking tokens and false otherwise.

Details:

  • What the server cannot see. A template takes what it reads: one that has no reasoning_effort renders the same at every level, and one that has no enable_thinking renders the same either way. The server cannot tell whether the template used an argument its renderer accepted. The MiniMax check establishes that the template prints a JSON declaration for every supplied tool in its own <tool>...</tool> block, under the tool's name. It does not read the template's instructions for how to call a tool: a template that prints such declarations and asks for another call format passes, and its calls come back as text. A variable the client sends in chat_template_kwargs is its instruction to the template. Qwen probes with marked tools before the real render, while MiniMax probes after a successful real render. The native Qwen check requires every marker it adds to the names and schemas to appear in the probe, and each real name to occur in the real prompt in literal or JSON-escaped form. If the probe passes, a tool that a template variable removes from the real declarations can still pass the name check when its name occurs elsewhere in the real prompt, such as in the conversation, another tool's name or the forced-choice instruction. Other spellings some clients use (reasoning, thinking, think) are not fields of this API and are ignored like any unknown field.
  • Rules 1 and 2 read enable_thinking as true or false. Only the effort none means no reasoning, ignoring case and surrounding whitespace. Recorded levels and mapped names are matched the same way. The request's own enable_thinking field is a boolean of the API on every tier: it is read as one when the request is parsed (a JSON 1 or "yes" arrives as true), before any of this applies. A non-null top-level switch overrides a nested switch. If the nested value survives, Qwen3.8-Flash-Next requires a boolean or returns a 400. With no recorded facts its type is preserved.
  • The server resolves the arguments once per request. Tool checks can render the conversation more than once. On the native path a recognized conversation refusal gives a 422 before any comparison. For other failures with request arguments, the server tries the render without them. If it succeeds, the server discards that prompt and returns a 400 naming the fields and quoting up to 300 characters of the template error. On the native vision-language path it reports the thinking-off default if it supplied it to the renderer with unrelated template arguments. If recorded facts changed the arguments, it shows up to 300 characters of the resolved arguments. If the comparison also fails, the server handles its error as it would without the fields. A recognized native conversation refusal gives a 422. A TypeError on a MiniMax text render with tools alone triggers explicit instructions with the request's arguments. For other TypeError, ValueError or Jinja template errors, both native and text paths return a 503 with the exception type and the first 300 characters of its message. Other exceptions propagate.
  • With minimax-m2 on a text template that takes tools, the server first renders an unforced request with the real tool names. After that succeeds, it renders the same conversation and arguments again with a unique identifier for each tool name. The server uses the real prompt only if the second render succeeds, contains every identifier and exactly reproduces the real prompt when the original names are put back. In the renamed render each name must be the name of a JSON declaration inside its own <tool>...</tool> block. Otherwise it renders explicit instructions with the real tools and the request's arguments. The parser reads only MiniMax's call format, so a template that presents tools another way (Qwen3-Coder's, for example) gets explicit instructions. A forced tool_choice always uses explicit instructions listing every supplied tool in request order, then directing the model to call the selected function. MiniMax also uses explicit instructions on vision launches and text launches without template tools. openai-json always uses instructions.
  • Explicit instructions list every supplied tool in request order, even when tool_choice selects one function. Changing this list between turns can shorten the reusable prefix or leave no usable vision checkpoint. When tools are enabled and the request supplies a nonempty tools list, the selected tool must be in that list. openai-json accepts only functions and lists them before the forced instruction. The server returns a 400 if it cannot prepare the instructions, if they omit a supplied tool or if their text is missing from the final prompt (differences in whitespace are accepted). Set tool_choice to none or launch with --tool-call-parser none to turn tools off.
  • chat_template_kwargs names must be nonempty. The launcher reserves names for renderer parameters, including tools, messages, tokenize, add_generation_prompt and chat_template. It also reserves the named parameters found in the renderer signature it inspects, except that enable_thinking and reasoning_effort are always allowed as names. Setting a reserved name gives a 400.

On oMLX

On oMLX the fields are oMLX's own. The launcher passes the request through unchanged, so what they do is upstream's behavior, and it is not the MLX runtime's (oMLX 0.7.3, Qwen3.8-Flash-Next):

The request sends MLX runtime oMLX
reasoning_effort at a native level, alone reasoning on at that level the effort is a template argument and nothing more: it does not turn reasoning on. With the shipped profile's default of thinking off, nothing changes (measured: 0 reasoning tokens for low, medium and xhigh sent alone). Send "enable_thinking": true with the effort
reasoning_effort none reasoning off alone or with enable_thinking false, 400 with true not an off switch: reasoning stays as enable_thinking or the default has it, and with reasoning on oMLX retries none as low (read in oMLX's source, not measured by us). Send "enable_thinking": false
an effort other than a native level or none minimal is sent as low, high as xhigh. Other values give a 400 retried under oMLX's own nearest name and, if the template refuses that too, rendered without the effort, with no error (read in oMLX's source, not measured by us)
the same name as a field and inside chat_template_kwargs a non-null top-level field wins for reasoning_effort, the one inside chat_template_kwargs wins (read in oMLX's source)

For the same reasoning setting on both runtimes, send enable_thinking true with a common native level (low, medium or xhigh), or send enable_thinking false alone for off. Keep top-level and nested fields consistent.

Measured on Qwen3.8-Flash-Next (E5 build, one 150-word writing prompt, temperature 0):

Request MLX runtime with the MTP drafter oMLX
thinking off (the default) 0 reasoning tokens, 4 s 0, 5 s
enable_thinking true, reasoning_effort low 2,250 reasoning tokens, 37 s 1,059, 35 s
the same, medium 2,809, 44 s 2,218, 66 s
the same, xhigh (also what thinking on without an effort gives) 1,821, 28 s 1,395, 42 s
reasoning_effort low, medium or xhigh alone the same prompt as with enable_thinking true, so the three rows above 0: reasoning stays off

Each level gave a different answer on this prompt. On both runtimes xhigh used fewer reasoning tokens than medium. The effort sets no separate reasoning budget. A positive max_tokens caps the whole answer.

Qwen3.8-Flash-Next with a drafter

Qwen3.8-Flash-Next has a multi-token-prediction (MTP) head. In the tests below, its drafter sped up generation and matched plain decoding at temperature 0. Set a positive max_tokens when comparing outputs. Which runtime can use the head depends on where it is:

The head is Runtime How
a separate drafter directory (python -m mlx_vlm.split_mtp, model type qwen4_exp_mtp), the usual case for MLX conversions, which strip it from the checkpoint MLX --draft-model <drafter directory>. The command below selects the MLX runtime and prints runtime=mlx (auto: model_type=qwen4_exp, --draft-model given, mlx only)
embedded in the checkpoint oMLX KAMIWAZA_MLX_OMLX_MODEL_SETTINGS='{"mtp_enabled": true}'. Refused if checkpoint metadata lists no matching tensor name (see "What a launch on oMLX does"). Not measured by us: no such checkpoint was at hand

The command for the first case, with the n-gram table kept on disk:

kamiwaza-mlx-server -m ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16 \
  --draft-model ./Qwen3.8-Flash-Next-PLEq8-E5-Dbf16-mtp \
  --ple-store ./e5-table.json

Measured through the server on a Mac Studio (M3 Ultra, 512 GB), E5 build, table on disk, temperature 0, generation rate as the server logs it:

Configuration Prose Code Accepted drafts (prose / code) Memory
oMLX, no drafter 38 tok/s 38 tok/s 91 GB
MLX runtime, no drafter 33 tok/s 33 tok/s 90 GB
MLX runtime, MTP drafter, default block (4) 53 tok/s 73 tok/s 55% / 88% 92 GB
the same, --draft-block-size 3 51 tok/s 64 tok/s 64% / 92% 92 GB
the same, --draft-block-size 5 52 tok/s 78 tok/s 46% / 82% 92 GB
the same, --draft-block-size 8 39 tok/s 74 tok/s 27% / 65% 92 GB
  • The output was the same text with and without the drafter, and at every block size, on all four test prompts (prose, code, and two tool-call turns with 25 tools).

  • The table stays on disk with the drafter on (the memory column).

  • Block size. The drafter's config.json says block_size: 2: that is the depth the head was trained at (one extra token). At run time mlx-vlm applies the head repeatedly, up to a block of 4 for this drafter when --draft-block-size is not given. Set at least 2 to allow one target token and at least one draft token per round. The block is a ceiling, not a fixed size: mlx-vlm drafts at the trained depth for the first rounds of a generation and goes up to the block only while at least 65% of recent rounds had their first draft accepted. 5 was a little faster on code and no faster on prose; 8 lost on prose, where most of a long draft is rejected. The default is a reasonable choice for mixed work.

  • Before the MLX runtime supported vision prompt reuse, a Mac Studio run on commit a05a122 with the Flash-Next E5 build took about 13–15 s per short-answer tool-calling turn with the MTP drafter. MLX kept the weights loaded and evaluated the full prompt (about 13,000 tokens, 25 tools) on each turn. On warm short-answer turns, oMLX without a drafter kept the prefix and answered in about 1–5 s. Long answers took longer on both runtimes.

  • On development commit 9174b27, the same Mac Studio and Flash-Next E5 build (n-gram table on disk) ran a scripted tool-calling conversation of ten steps with a long system prompt and 25 tools. The prompt grew from 13,691 to 18,143 tokens. At each step we timed two complete responses at temperature 0 without streaming. The second request repeated the first request's prompt, so with reuse on it could restore a checkpoint the first request had just stored. At steps 3 and 7 the prompt crossed a boundary. With reuse on, the second request restored 2,048 more tokens and finished sooner than the one-token request at both steps. The first two rows give the minimum / median / maximum for steps 2 to 10. We measured process footprint at the end of each configuration, after the scripted conversation for reuse off and after the additional tool-call phases for reuse on.

    Reuse off, no drafter Reuse on, no drafter Reuse off, MTP drafter Reuse on, MTP drafter
    One-token request (taken as time to first token) 14.1 / 15.8 / 17.9 s 1.06 / 1.95 / 2.74 s 13.7 / 15.6 / 17.6 s 1.03 / 1.92 / 2.71 s
    The same prompt again, up to 400 tokens 14.4 / 16.4 / 18.4 s 1.17 / 2.30 / 2.67 s 14.2 / 16.1 / 18.1 s 1.00 / 1.93 / 2.43 s
    First prompt of the conversation, one token 62.0 s, including the first read of the weights 16.2 s 13.6 s 13.7 s
    Prompt tokens restored none 12,288, then 14,336, then 16,384 none 12,288, then 14,336, then 16,384
    Final process footprint 91 GB 91 GB 93 GB 93 GB

    The assistant messages matched with reuse on and off at all ten steps for both requests, without the drafter and with it (40 comparisons). They also matched with and without the drafter. Both comparisons left out the tool-call IDs the server generates per response. The first request in each configuration started with no checkpoint and read the full prompt. With reuse on, each later request of this conversation read on from the last kept 2,048-token boundary. The two checkpoints kept took about 0.5 GB each. Reuse depends on matching prompt tokens. A new conversation can share a kept prefix, and an edit to the history can leave no matching checkpoint. Commit 9174b27 also returned a tool call written after prose as a call. Every answer compared here was plain text without a tool call, and the prompts were scripted, with no answer fed back, so that change did not shape the messages compared. The times were taken with that commit's parser in the response path.

  • On commit a05a122, oMLX returned valid calls at its server defaults on all 60 repetitions of the short two-tool weather request, 30 without streaming and 30 with streaming. At the launcher's old MLX defaults (temperature=1.0, top_p=1.0, no top-k), MLX made 59 valid calls without a drafter and 54 with the MTP drafter. Sending "top_p": 0.95 brought the drafted run to 60 of 60. The code test sent ten prompts twice each, with 25 tools and about 13,000 input tokens per request. oMLX returned 20 valid calls. MLX returned 19 without a drafter and 18 with the MTP drafter at the launcher's old defaults. The MTP run with top_p=0.95 also returned 18. All 25-tool misses came from one request sent twice per run. On each miss the model answered with a sentence followed by the call. These requests used automatic tool choice, so the server returned that answer as text. In the separate temperature-zero probes, every tool-call turn tested on oMLX without a drafter and on MLX without a drafter or with the MTP drafter returned a valid call.

  • On the MLX runtime, ordinary streaming sends reasoning in content. A generation limit can end the answer before </think>. The --strip-thinking option or strip_thinking: true can filter paired tags and their contents. With auto-detection on, one-piece answers, text streams and vision-language streams start inside a think block if the request's rendered prompt ends with an open <think> tag, and outside one if it ends with </think>. Think tags earlier in the prompt are not read. Some renderers return the last message itself as the prompt (mlx-vlm's helper does for PaliGemma, Florence-2 and Falcon OCR); when the prompt ends with the text of the request's last message (where the message keeps its parts, its last text part that is not blank) and that text ends with the tag after text of its own, the tag is treated as the client's text and is not read either (a heuristic: the renderer does not say who wrote the end of the prompt). A prompt that ends with neither uses the process-wide detection result, which includes the start-up probe of the template rendered without request arguments. When a block was inferred from the prompt and the output has no think tag at all, an output of 50 tokens or fewer is withheld and a longer one is returned as the answer. On a stream without tools, a block whose opening tag the filter recognizes and that does not close is not sent in content. A one-piece answer (non-streamed, or streamed with tools) is split by the same rules as before: when its first think tag is </think>, the text after that tag, without leading newlines, is the content, think tags included; otherwise paired blocks are removed, and an unmatched tag and the text after it can stay in content. The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model. With tools enabled the runtime generates the answer in one piece and returns any extracted reasoning as reasoning_content. oMLX sends reasoning as reasoning_content.

  • A DFlash 2 drafter for this model (incoai/Qwen3.8-Flash-Next-DFlash2, revision a22a1744) loads with mlx-vlm 0.7.4, but drafting fails on the model's hidden-state layout. The start-up self-test requires a completed draft round and exits 2 with mlx-vlm's error when generation raises. oMLX's DFlash does not cover this model type either.

Native speculative decoding (MTP)

Vision-language checkpoints loaded through mlx-vlm can use mlx-vlm's own speculative decoding with a separate drafter: a model's native MTP (multi-token prediction) head, a DFlash2 drafter, or an EAGLE-3 drafter. In greedy tests with the same positive token limit, Qwen3.8-Flash-Next produced the same output with and without its MTP drafter. Many MLX conversions strip the MTP head, so extract it from a checkpoint that still carries it:

python -m mlx_vlm.split_mtp --model ./Qwen3.8-Flash-Next-FP8 --output ./Qwen3.8-Flash-Next-MTP

python -m kamiwaza_mlx.server \
  -m ./Qwen3.8-Flash-Next-mlx \
  --draft-model ./Qwen3.8-Flash-Next-MTP
Flag Env var Default Meaning
--draft-model KAMIWAZA_MLX_DRAFT_MODEL off Drafter path or HF id.
--draft-kind KAMIWAZA_MLX_DRAFT_KIND auto auto, mtp, dflash or eagle3; auto reads the drafter's config.
--draft-block-size KAMIWAZA_MLX_DRAFT_BLOCK_SIZE mlx-vlm's default for the drafter (4 for the Qwen3.8-Flash-Next MTP head) Maximum tokens per round, including one target token. The minimum is 2. With 1, mlx-vlm drafts nothing and ends every answer after its first token. The Qwen3.8-Flash-Next MTP head uses shorter rounds when too few recent rounds accept the first draft token.
--draft-queue-timeout KAMIWAZA_MLX_DRAFT_QUEUE_TIMEOUT 600 Seconds a request waits for the drafter before failing with 503 and Retry-After: 5; 0 waits indefinitely. The wait ends when the request takes the drafter. Parsing the request and checking its token limit and reasoning fields come before the wait; loading its images and building its prompt come after.
--draft-stream-send-timeout KAMIWAZA_MLX_DRAFT_STREAM_SEND_TIMEOUT 60 Seconds a draft-enabled stream may wait for the client to accept one chunk before it is aborted and the drafter released; 0 disables.
  • With a native drafter, an unset, null, zero or negative token limit becomes the served context window minus the token count of the prompt text, with a minimum of 1. Image tokens are excluded, so an image request can get more room than the window has left. If the window is unknown, the limit is effectively unbounded (2**31 - 1). A positive limit and every limit on a launch without a native drafter go to generation as before. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens. On the kamiwaza-mlx-server chat endpoint on the MLX runtime, max_tokens and max_completion_tokens name this same limit. Send either or both when equal. Null for either means unset. Different integers give a 400, for example max_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields. The vision path does not report finish_reason: length, so a text answer that reaches the substituted limit reports stop.
  • Startup fails if the drafter cannot pair with the target model, if the model loads through mlx-lm (for example with --text-only), or if --dflash-draft-model is also set. The server starts serving only after the requested drafter completes a round in its self-test.
  • The drafter serves one generation at a time, so draft-enabled requests queue. A streamed response holds the drafter until its last chunk is sent, so a slow reader delays everyone behind it (each chunk must be accepted within --draft-stream-send-timeout, or the stream is aborted). --draft-queue-timeout bounds the wait with a 503, but the timer runs on the server's event loop: while a non-streamed or tool-call generation is running it cannot fire, so a waiter may be answered (or rejected) only after that generation finishes. A client that disconnects while queued is skipped when its turn comes, on a best-effort basis (a disconnect the server has not yet read is not detected). A request that completes a draft round logs the mean tokens per round (one target token plus the accepted drafts) and the fraction of drafted tokens accepted.
  • Startup also fails if the hidden size or vocabulary size the launcher reads from the drafter's configuration (its text configuration first, then the top level) differs from the target model's (for example a drafter built for another model family; a drafter that records the target's width as backbone_hidden_size or target_hidden_size is compared by that value; sizes recorded elsewhere, such as an EAGLE-3 drafter's vocabularies, are not compared), or if an explicit --draft-kind disagrees with the drafter's own type.
  • After loading the drafter, the server tries up to three greedy text generations of up to eight tokens each. A generation passes when it finishes with at least one completed draft round, even if the target rejected every proposal. If generation raises or all three finish without a round, the launch exits 2 with the reason. The test warms the weights its prompts use. The log line starting with Drafter self-test passed in reports the elapsed time and completed rounds. The earlier single-generation test took 3 to 13 s for Qwen3.8-Flash-Next in our runs, depending on the system's file cache. Sampled decoding, image inputs and the HTTP layer are outside this test.
  • Successful non-streamed responses for requests that complete a draft round include X-Speculative-Tokens-Per-Round, X-Speculative-Acceptance-Rate and X-Speculative-Rounds. A request limited to one token has none of these headers.
  • Measured through the server on Qwen3.8-Flash-Next with its MTP head (M5 Max, 128 GB, temperature 0): 37 tok/s without a drafter; 52 tok/s on prose and 73 tok/s on code with it. Gains are smaller with sampling (temperature 1.0: 36 and 62 tok/s).

N-gram table on disk (Qwen3.8-Flash-Next)

Qwen3.8-Flash-Next (mlx-vlm model type qwen4_exp) carries a very large per-layer n-gram embedding table, also called the PLE or "engram" table, of about 51 billion values. It is a lookup table. Loaded as ordinary parameters it occupies tens of gigabytes of unified memory; mlx-vlm can instead memory-map the table inside the checkpoint's safetensors files and read only the rows each token needs.

A model directory opts in through text_config.ple_storage in its config.json (mlx-vlm's prepare_external_ple_model writes such a directory). --ple-store makes the same choice at launch, without preparing a second directory, and can point at the table of another conversion of the same model:

# Index the table of a converted checkpoint in place (writes a small JSON file;
# no weights are copied). The manifest refers to the checkpoint by relative
# path, so keep the two where they are.
python -m kamiwaza_mlx.ple_store ./Qwen3.8-Flash-Next-mlx-8bit ./tables/q8/ple-store.json

# Serve 4-bit weights with that 8-bit table read from disk.
python -m kamiwaza_mlx.server \
  -m ./Qwen3.8-Flash-Next-mlx-4bit \
  --ple-store ./tables/q8/ple-store.json
Flag Env var Default Meaning
--ple-store KAMIWAZA_MLX_PLE_STORE off Manifest of the n-gram table to read from disk. Replaces the table in the model directory, and a manifest the model's config.json declares.
  • The table bundled in the model directory is not loaded when --ple-store is set. To keep a model's own table on disk, index that same directory and pass the manifest.
  • Use a table from a conversion of the same model. The launcher can only check its shape: a table with a different number of rows or row width is refused, but a table of the same shape from another model is not detected and would give wrong output. Supported table layouts are 4-bit and 8-bit affine with group size 32, and 4-bit nvfp4. Anything else fails at startup with the reason.
  • Startup fails, and the server never serves without the requested table, if the manifest is missing or incomplete, if the model is not a qwen4_exp model, if --text-only or --dflash-draft-model is also set, if the table does not fit the model, or if the installed mlx-vlm has no on-disk table support (the pinned 0.7.4 has it). A table that does not fit is reported as the n-gram table of --ple-store <manifest> could not be used: <reason>; a failure of the model itself keeps its own message.
  • The model must be one this server loads through mlx-vlm, which is what it does for a conversion that keeps its vision tower. For any other model startup fails; it is not forced through mlx-vlm.
  • Lookups read from the page cache, so the first tokens after a cold start are slower. --cache-rows N on the manifest command records a row cache size in the manifest: the N most recently used rows are kept in memory. With --ple-store the manifest's value is the one used, whatever the model directory's own config.json declares.

Experimental DFlash speculative decoding

DFlash is available as an opt-in text-only generation path. Configure a draft model and the server will use a DFlash speculative loop instead of the normal mlx_lm.generate.stream_generate loop. The default backend is auto: it uses the standalone dflash-mlx implementation when installed, otherwise it falls back to ZLab's dflash.model_mlx.

With --strip-thinking or strip_thinking: true, the stream of this path decides per request whether an answer starts inside a think block (see --strip-thinking under "Most useful flags"). That rule is covered by unit tests only on this path; it was not run with a live model.

The DFlash backends install from git (PyPI does not allow direct-URL dependencies in package metadata). These are the tested pins:

# ZLab backend
pip install "dflash[mlx] @ git+https://github.com/z-lab/dflash.git@6d6229eaddce58fcd7e4cc91945da632f5544c86"

# Standalone dflash-mlx backend
pip install "dflash-mlx @ git+https://github.com/bstnxbt/dflash-mlx.git@fada1eb2b75cd1c875ca6547b6518783fd3d2956"

python -m kamiwaza_mlx.server \
  -m mlx-community/Qwen3.5-27B-5bit \
  --dflash-draft-model z-lab/Qwen3.5-27B-DFlash \
  --dflash-backend auto \
  --dflash-block-size 16 \
  --dflash-default-max-tokens 2048 \
  --dflash-max-context -1

The first five environment variables correspond to the DFlash options above. The last three explicitly set verification and stream synchronization to their defaults.

export KAMIWAZA_MLX_DFLASH_DRAFT_MODEL=z-lab/Qwen3.5-27B-DFlash
export KAMIWAZA_MLX_DFLASH_BACKEND=auto
export KAMIWAZA_MLX_DFLASH_BLOCK_SIZE=16
export KAMIWAZA_MLX_DFLASH_DEFAULT_MAX_TOKENS=2048
export KAMIWAZA_MLX_DFLASH_MAX_CONTEXT=-1
export KAMIWAZA_MLX_DFLASH_VERIFY_MODE=auto
export KAMIWAZA_MLX_DFLASH_VERIFY_QMM=true
export KAMIWAZA_MLX_DFLASH_STREAM_SYNC_EVERY=0

Notes:

  • DFlash is only enabled when --dflash-draft-model is non-empty, and that forces a text-only mlx-lm target load even for Qwen3.5 conversions that retain a vision_config.
  • The Qwen3.5-27B drafter is trained for target Qwen/Qwen3.5-27B; MLX quantized conversions such as mlx-community/Qwen3.5-27B-5bit and mlx-community/Qwen3.5-27B-6bit are structurally compatible targets to try, but acceptance and quality need measurement per quant.
  • The z-lab/Qwen3.5-27B-DFlash draft config uses five full_attention layers. The dflash-mlx runtime has draft sink/window machinery for other draft shapes, but those knobs do not change this Qwen3.5 speculative model's draft attention.
  • Server KV and prefix caches are disabled while DFlash is enabled because DFlash allocates its own per-request target/draft prompt caches and mutates target model hooks during generation.
  • --dflash-backend auto picks dflash-mlx when the standalone dflash_mlx package is installed (see the tested git pin above); otherwise it falls back to the ZLab backend. The inline dflash-mlx adapter runs without that package's own prefix snapshot cache, so it is meant for local kernel/runtime experiments, not as a replacement for that package's full OpenAI-compatible server. It also installs a Qwen target-op compatibility patch that skips the full-vocabulary lm-head projection for large prompt-prefill chunks whose logits are discarded by the DFlash loop, and disables unused prefix-snapshot work because the inline adapter does not provide a dflash-mlx prefix cache.
  • For eligible 4-bit or 8-bit MLX target linears, the dflash-mlx adapter also installs that package's verifier-linear wrappers. Use --dflash-verify-mode off to disable the swap, or --no-dflash-verify-qmm to keep the wrappers while disabling the custom qmm kernels.
  • --dflash-max-context is a degradation guard: prompts at or above that token count bypass DFlash and use the normal MLX generation path. Omit it, or set it to -1, to use the conservative auto cap of 4000 prompt tokens. Set it to 0 only when you intentionally want to route every prompt through DFlash. Because DFlash mode disables the server-level shared KV/prefix caches, bypassed requests do not use shared prefix-cache optimizations, but they still allocate the normal bounded per-request generation cache.
  • DFlash generation is serialized with an internal lock. That protects the target model hidden-state hooks and Qwen3.5 gated-delta rollback patch.
  • Extra mx.synchronize() calls in the DFlash streaming loop are disabled by default. Set --dflash-stream-sync-every N only when diagnosing stream timing or device scheduling.

Note: Not specific to Kamiwaza (that is, you can use on any Mac, Kamiwaza not required)

pip install kamiwaza-mlx

# start the server
a) python -m kamiwaza_mlx.server -m ./path/to/model --port 18000
# or, if you enabled the optional entry-points during install
b) kamiwaza-mlx-server -m ./path/to/model --port 18000

# chat from another terminal (note: specify --host to match server port)
python -m kamiwaza_mlx.infer --host localhost:18000 -p "Say hello"

The remainder of this README documents the original features in more detail.

MLX-LM 🦙 — Drop-in OpenAI-style API for any local MLX model

A FastAPI micro-server (server.py) that speaks the OpenAI /v1/chat/completions dialect, plus a tiny CLI client (infer.py) for quick experiments. Ideal for poking at huge models like Dracarys-72B on an M4-Max/Studio, hacking on prompts, or piping the output straight into other tools that already understand the OpenAI schema.


✨ Highlight reel

Feature Details
🔌 OpenAI compatible Same request / response JSON (streaming too) – just change the base-URL.
📦 Zero-config Point at a local folder or HuggingFace repo (-m /path/to/model).
🖼️ Vision-ready Accepts {"type":"image_url", …} parts & base64 URLs – works with Qwen-VL & friends.
🎥 Video-aware Auto-extracts N key-frames with ffmpeg and feeds them as images.
🧮 Usage metrics Prompt / completion tokens + tokens-per-second in every response.
⚙️ CLI playground infer.py gives you a REPL with reset (Ctrl-N), verbose mode, max-token flag…

🚀 Running the server

# minimal
python -m kamiwaza_mlx.server -m /var/tmp/models/mlx-community/Dracarys2-72B-Instruct-4bit

# custom port / host
python -m kamiwaza_mlx.server -m ./Qwen2.5-VL-72B-Instruct-6bit --host 0.0.0.0 --port 12345

Default host/port: 0.0.0.0:18000

Most useful flags:

Flag Default What it does
-m / --model mlx-community/Qwen2-VL-2B-Instruct-4bit Path or HF repo.
--host 0.0.0.0 Network interface to bind to.
--port 18000 TCP port to listen on.
-V / --vision off Force vision pipeline; otherwise detect declared vision config.
--text-only off Explicit language-only load for converted checkpoints retaining vision metadata; mutually exclusive with --vision.
--strip-thinking off Removes <think>…</think> blocks from model output. A request can set strip_thinking: true. With auto-detection on, one-piece answers, text streams and vision-language streams start inside a think block if the request's rendered prompt ends with an open <think> tag, and outside one if it ends with </think>. Think tags earlier in the prompt are not read, nor a tag that ends the text of the request's last message (where the message keeps its parts, its last text part that is not blank) where the prompt ends with that text and the text has more of its own before the tag (a heuristic for renderers that return the last message as the prompt). A prompt that ends with neither uses the process-wide detection result, which includes the start-up probe of the template rendered without request arguments. The DFlash stream path of this rule is covered by unit tests only; it was not run with a live model.
--draft-model off Native mlx-vlm speculative drafter (MTP, DFlash2, EAGLE-3); see Native speculative decoding.
--vlm-prompt-reuse auto Reuse matching prompt prefixes on eligible MLX vision-language models for requests without images or media placeholders. See Prompt reuse for vision-language models.
--sampling-defaults model Choose defaults for sampling fields omitted from a request. See Sampling defaults.
--enable-prefix-caching True Enable automatic prompt caching for text-only models. If enabled, the server attempts to load a cache from a model-specific file in --prompt-cache-dir. If not found, it creates one from the first processed prompt and saves it.
--prompt-cache-dir ./.cache/mlx_prompt_caches/ Directory to store/load automatic prompt cache files. Cache filenames are derived from the model name.

System-prefix cache (system-only)

Flag Default What it does
--enable-prefix-caching True Enable system-prefix caching for text-only models.
--prompt-cache-dir ./.cache/mlx_prompt_caches/ Directory for system-prefix cache files (.safetensors, .len, .hash).
--system-cache-max-tokens 2048 Max tokens to cache from the system prompt (0 = unlimited). If the system prompt exceeds this cap, system caching is skipped.
--prefix-cache-headroom 64 Extra tokens reserved beyond the system prompt length when sizing the system cache.

Conversation cache (simple global prefix-match)

Simple mode uses a single text-model conversation cache. If the previous prompt is a prefix of the incoming prompt, the server skips prefill for that prefix. Otherwise it resets the cache. Conversation IDs appear in logs and metadata.

Flag Default What it does
--disable-kv-cache off Disable the launcher's retained KV caches, including vision prompt checkpoints. Generation still uses temporary KV state within each request.
--kv-cache-max-tokens 0 Per-cache upper bound (0 = min(model context, 128k)).
--kv-cache-keep 4 Tokens to keep when trimming rotating caches.
--kv-cache-idle-release-seconds 0 Release KV caches, including retained vision prompt checkpoints, on the next chat request after the idle interval (0 disables idle release).
--kv-cache-hard-reserve True Fully reserve KV memory up to sizing target.
--kv-cache-warmup False Run a warm-up pass at startup to materialize KV shapes.
--kv-cache-warmup-tokens 0 Warmup tokens (0 = use resolved target size).
--retain-mx-cache False Keep MX allocator memory (disables mx.clear_cache).

Prompt reuse for vision-language models

mlx-vlm can save the target model's prompt state at a 2,048-token boundary. An eligible request can continue from the longest retained prefix that matches token for token. Prompts of 2,048 tokens or fewer are always read in full. For 2,049 tokens, the end-minus-one checkpoint falls exactly at 2,048 and can be kept. The alignment check rejects an end-minus-one checkpoint only when it falls off the grid. Even an identical prompt leaves its final token to read.

Requests with images or a media placeholder such as <|image_pad|> anywhere in the rendered prompt bypass reuse. Either can cause mlx-vlm 0.7.4 to split a prefill chunk at a different token, even on a cache miss. When the input IDs contain no media placeholders, chunked prefill uses mlx-vlm's default step, and the request uses MTP or no drafter, restoring a grid boundary preserves the forward calls for the remaining prompt. The offline Metal test compares generated tokens with and without reuse at temperature 0 (greedy decoding) on tiny random Qwen weights for repeated prompts, growing conversations and shared-prefix forks.

Each running request that restores a checkpoint uses its own copy. Reuse adds no new request-wide lock. The manager takes its own lock around storing and looking up checkpoints.

Flag Environment variable Default Meaning
--vlm-prompt-reuse {auto,on,off} KAMIWAZA_MLX_VLM_PROMPT_REUSE auto Enable reuse when the startup checks pass. Use on to require the manager at startup or off to disable it.
--vlm-prompt-reuse-checkpoints N KAMIWAZA_MLX_VLM_PROMPT_REUSE_CHECKPOINTS 2 Keep at most N checkpoints in memory across requests. The minimum is 2 because mlx-vlm reserves one slot when choosing up to N−1 intermediate boundaries per request.
--vlm-prompt-reuse-max-gb G KAMIWAZA_MLX_VLM_PROMPT_REUSE_MAX_GB 8 Maximum total size of retained checkpoint cache buffers in GiB. Token lists and bookkeeping are outside this limit. The value converted to bytes must be finite and at least one byte.

Blank or whitespace-only environment values use the defaults above. The MLX runtime validates both numeric settings at startup in every mode, including off. Reuse requires KV caching and a listed vision-language model type. The list currently contains only qwen4_exp (Qwen3.8-Flash-Next). See config/prompt_reuse.json.

The eligibility check runs after the model and any vision drafter have loaded and the drafter's self-test has passed. It accepts MTP or no drafter. DFlash and EAGLE-3 need hidden states for the entire prompt. The launcher verifies that the tokenizer resolves each configured media token ID to a placeholder string and that tokenizing a plain, media-free probe produces none of those IDs. It builds the manager, checks the whole-model checkpoint plan and the model's cache layout, and exercises the manager's statistics, byte accounting and clear operations. If reuse is unavailable, auto leaves it off and on exits 2. Starting successfully with on guarantees that these checks passed. A checkpoint can still be too large to retain. A cap of one byte stores no checkpoints.

Conversation IDs and reset_conversation apply to the text-model conversation cache. The first vision request in a new conversation can restore a kept prefix if its tokens match, for example when it starts with the same system prompt and tools. This also applies with reset_conversation: true. If an edit to the history leaves no matching checkpoint, the next request reads the full prompt. To force the vision model to read every prompt in full, start the MLX runtime with --vlm-prompt-reuse off.

The launcher decides whether to create the manager and keeps it in memory. It constructs the manager directly, bypassing the mlx-vlm factory that reads APC_ENABLED, APC_DISK_ENABLED, APC_DISK_PATH, APC_BLOCK_SIZE and APC_NUM_BLOCKS. These variables have no effect on vision prompt reuse. The launcher supplies the settings for APC_CHECKPOINT_ENTRIES and APC_MEMORY_MAX_GB from the two numeric options. It sets the guard (APC_CHECKPOINT_GUARD_TOKENS) to 1 token, the interval (APC_CHECKPOINT_INTERVAL_TOKENS) to the default prefill step and the reserve (APC_MEMORY_RESERVE_GB) to mlx-vlm's automatic value. The manager reads some APC_* settings directly. For example, the launcher caps a valid APC_EXACT_MIN_TOKENS at one prefill step. A nonnumeric value makes the startup check fail. The launcher reports mlx-vlm's error text and appends the names of inherited APC_* variables. The error text itself can include a value.

Retained checkpoints are evicted least recently used first. Under memory pressure mlx-vlm can also evict a checkpoint or skip a store or restore. Even for a single Qwen request, mlx-vlm makes room before extracting a checkpoint from the prompt cache. Under pressure it can evict a matching prefix at that step, even if the launcher then declines to store an off-grid end-of-prompt checkpoint. If all matching prefixes were evicted, the next replay reads the whole prompt.

A successful non-streamed vision response reports the restored prefix length in usage.prompt_tokens_details.cached_tokens, including zero on a miss or bypass. For a streamed request, the per-request restore count is available only in the generation log. The X-Conv-* headers describe the text conversation cache. With reuse enabled, the launcher logs the restored and total prompt tokens for each eligible generation that produced a result. The log records outcome=completed, outcome=abandoned or outcome=failed, even if the generation ended before its first result. The launcher logs a bypass reason for images or media placeholders. A warning records an increase in the manager's memory skip or memory eviction counter only when the snapshots before and after the request both report a nonnegative integer for that counter (booleans are invalid).

GET /v1/kv_cache/stats reports the startup decision and limits in vlm_prompt_reuse. checkpoints is the configured entry limit. resident_bytes is the total size of the kept checkpoints' cache buffers. It excludes copies in use by running requests and is null when reuse is off. If the statistics read fails or a required field is missing or is not a nonnegative integer (booleans are invalid), the block includes statistics_error, and resident_bytes and mlx_vlm_checkpoint_counters are both null.

  • launcher_counters counts eligible requests and tracks image and media placeholder bypasses separately. restored_requests counts eligible requests whose generation result reported a restored prefix. If mlx-vlm restores a prefix and generation fails before its first result, the launcher counts the request in eligible_requests and eligible_without_result. Only mlx-vlm records the restore in that case, so its counters can differ from the launcher's. The launcher's prompt-token and restored-token totals cover eligible requests that produced a result. The launcher keeps its counts for the lifetime of the process.
  • mlx_vlm_checkpoint_counters reports the manager's checkpoint stores and hits, restored tokens, memory skips, memory evictions, evictions and rejects. Ordinary replacement at the entry limit does not increment mlx-vlm's eviction counter. The manager's counters restart at zero when it clears the checkpoints, including on idle release. If the idle clear raises an exception, the launcher warns that checkpoints may have been kept and continues serving the request.

totals.kv_bytes includes the resident checkpoint bytes. It is null when statistics_error is present, because the checkpoint bytes are then unknown.


Experimental multi-node via mlx.distributed

This section documents the direct server.py distributed bootstrap path.

It is distinct from the current Kamiwaza-managed two-node serving flow, which uses kamiwaza-mlx-distributed plus upstream mlx_lm.server and is documented in MLX_TWO_NODE_README.md.

The server can bootstrap a two-node mesh using mlx.distributed. Set a rendezvous host via PAIRED_HOST (optionally in a .env file) and launch each node with matching ranks/world-size. The helper will automatically read .env files passed via --distributed-env-file or located beside the server script.

# shared settings (either export or place in .env)
PAIRED_HOST=10.0.0.2
PAIRED_PORT=17863
WORLD_SIZE=2

# leader node (rank 0 hosts FastAPI)
RANK=0 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model

# worker node (rank 1 participates in mlx.distributed but does not bind HTTP)
RANK=1 python -m kamiwaza_mlx.server --distributed-env-file .env -m ./model

Useful knobs:

  • --distributed – force-enable/disable distributed mode (auto when a paired host is set, or PAIRED_WORLD_SIZE / MLX_DISTRIBUTED_WORLD_SIZE / --distributed-world-size is above 1; a bare WORLD_SIZE does not turn it on).
  • --distributed-rank / --distributed-world-size – override RANK/WORLD_SIZE env vars.
  • --distributed-host / --distributed-port – override PAIRED_HOST / PAIRED_PORT.
  • --distributed-server-rank – choose which rank should host the HTTP server (defaults to 0).

In this direct path, non-leader ranks simply keep the MLX runtime alive for collective ops once the model weights are synchronized. Treat it as a bootstrap/experimentation path, not as the primary Kamiwaza-managed TP=2 serving mode.


💬 Talking to it with the CLI

python kamiwaza_mlx/infer.py --host localhost:18000 --max_new_tokens 2048

Interactive keys

  • Ctrl-N: reset conversation
  • Ctrl-C: quit

🌐 HTTP API

GET /v1/models

Returns a list with the currently loaded model:

{
  "object": "list",
  "data": [
    {
      "id": "Dracarys2-72B-Instruct-4bit",
      "object": "model",
      "created": 1727389042,
      "owned_by": "kamiwaza"
    }
  ]
}

The created field is set when the server starts and mirrors the OpenAI API's timestamp.

POST /v1/chat/completions

{
  "model": "Dracarys2-72B-Instruct-4bit",
  "messages": [
    { "role": "user",
      "content": [
        { "type": "text", "text": "Describe this image." },
        { "type": "image_url",
          "image_url": { "url": "data:image/jpeg;base64,..." } }
      ]
    }
  ],
  "max_tokens": 512,
  "stream": false
}

On the kamiwaza-mlx-server chat endpoint on the MLX runtime, max_completion_tokens and max_tokens name the same limit. Send either field, or both with equal values. A null for either field means it is unset. Different integers return HTTP 400, for example max_tokens=30 and max_completion_tokens=50 differ. Send one of the two fields.

With a native drafter, an unset, null, zero or negative limit becomes the served context window minus the token count of the prompt text, with a minimum of 1. Image tokens are excluded, so an image request can get more room than the window has left. An unknown window gives an effectively unbounded limit of 2**31 - 1. A positive limit and every limit on a launch without a native drafter go to generation as before. With a native drafter the server treats zero as unset. On a vision-language launch without one it passes zero through to mlx-vlm; for an autoregressive model its generation then emits no tokens. The vision path does not report finish_reason: length, so a text answer that reaches the substituted limit reports stop.

Response (truncated):

{
  "id": "chatcmpl-d4c5…",
  "object": "chat.completion",
  "created": 1715242800,
  "model": "Dracarys2-72B-Instruct-4bit",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The image shows…" },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 143,
    "completion_tokens": 87,
    "total_tokens": 230,
    "tokens_per_second": 32.1
  }
}

Add "stream": true and you'll get Server-Sent Events chunks followed by data: [DONE].

System Prefix Caching (Text-Only Models):

  • Purpose: Dramatically speed up repeated queries that share the same system context (e.g., large document in role: system). The server caches only the system message(s), not the whole prompt, so subsequent turns process only new user tokens.
  • Flags:
    • --enable-prefix-caching (default True)
    • --prompt-cache-dir (default ./.cache/mlx_prompt_caches/)
    • --system-cache-max-tokens (default 2048, 0 disables the cap)
    • --prefix-cache-headroom (default 64)
  • How it works (high‑level):
    1. On first request with a system message, the server builds a KV cache for just the system portion and saves three files under --prompt-cache-dir:
      • <model>.safetensors (KV), <model>.safetensors.len (token count), <model>.safetensors.hash (SHA256 over token IDs)
    2. On subsequent requests with the same system text (hash matches), the server deep‑copies the cached KV and processes only new user/assistant tokens.
    3. If the system message changes, the old cache is discarded and replaced automatically.
    4. If the system prompt exceeds --system-cache-max-tokens, system caching is skipped and the full prompt is prefetched normally.
  • Example: A 10,000‑token system document is processed once; later questions only process the user tokens.
  • Notes: text‑only models; fully transparent to clients (no special fields needed).

Conversation KV Caching (Long chats, fast follow‑ups):

  • Rationale: Reuse KV across turns so only the tail of the prompt is prefetched.
  • Behavior:
    • Provide conversation or conversation_id (or X-Conversation-Id) if you want IDs reflected in logs/metadata. If omitted, the server uses default.
    • Successful JSON and SSE responses include the text conversation cache headers:
      • X-Conv-Id (resolved ID), X-Conv-KV (cache mode), X-Conv-Cached-Tokens, X-Conv-Processing-Tokens.
    • Non‑stream JSON also includes usage.input_tokens_details.cached_tokens and metadata.conversation_id.
    • The global cache is capped at min(model context, 128k) unless overridden by --kv-cache-max-tokens.
    • Only one request at a time uses the text conversation cache. Concurrent text requests that bypass a busy cache report X-Conv-KV: busy. Vision requests with KV caching enabled report X-Conv-KV: vision, including when they restore a prompt checkpoint.
  • Breaking change: the legacy /v1/conv_kv/* endpoints were removed in this branch.

🛠️ Internals (two-sentence tour)

  • server.py – loads the model with mlx-vlm, converts incoming OpenAI vision messages to the model's chat-template, handles images / video frames, and streams tokens back. For text-only models, if enabled via server flags, it automatically manages a system message cache to speed up processing when multiple queries reference the same system context.
  • infer.py – lightweight REPL that keeps conversation context and shows latency / TPS stats.

That's it – drop it in front of any MLX model and start chatting!

Metadata

Release files for kamiwaza-mlx 0.2.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kamiwaza-mlx 0.2.9
File Size Uploaded Provenance Actions
kamiwaza_mlx-0.2.9.tar.gz 515.9 kB Without provenance Details for kamiwaza_mlx-0.2.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kamiwaza-mlx 0.2.9
File Interpreter ABI Platform Actions
kamiwaza_mlx-0.2.9-py3-none-any.whl Python 3 none any Details for kamiwaza_mlx-0.2.9-py3-none-any.whl

Total release size: 765.1 kB

Release files / kamiwaza_mlx-0.2.9.tar.gz

Download URL kamiwaza_mlx-0.2.9.tar.gz
Size 515.9 kB
Tags Source
SHA-256 checksum
How to use checksums
58fecd0c6f838bcdd7c582811cab459a7ea46d962d09689caef9dfca479a9e84
BLAKE2b-256 checksum
How to use checksums
866c1630d6c5dcc68da9710f45739e119053fb456185972398f5e2f17571afad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.10.16

Release files / kamiwaza_mlx-0.2.9-py3-none-any.whl

Download URL kamiwaza_mlx-0.2.9-py3-none-any.whl
Size 249.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
42a7db836a3257fdc7a83d55fb36f61b2ac28b758ec44b4ffbb0342d2ba12f41
BLAKE2b-256 checksum
How to use checksums
78c2ae2e0556d55c6781ceb93736bfb234fcf4ce1022fb3ef6605bf2b960d486
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.10.16

Security for kamiwaza-mlx 0.2.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page