MLX GPT-OSS Server

Minimal OpenAI-compatible server for GPT-OSS/Harmony models on Apple Silicon.
Built with mlx-lm (inference), openai-harmony (prompt formatting), and FastAPI (HTTP API).

Feature List

• OpenAI-style /v1/chat/completions endpoint
• OpenAI-style /v1/responses endpoint
• Streaming (SSE) and non-streaming responses
• Harmony reasoning_effort support (low, medium, high)
• OpenAI tool-calling response format
• Responses API function-calling and previous_response_id support
• Robust Harmony tool-calling parser and stream recovery paths
• Usage token counts in responses
• /health queue stats and /v1/models compatibility endpoint
• Single-model runtime with FIFO request queueing

Requirements

• macOS on Apple Silicon
• Python >=3.11

Quick Start

pip install mlx-gpt-oss
mlx-gpt-oss --model mlx-community/gpt-oss-20b-MXFP4-Q8

Default bind: http://0.0.0.0:8000

Install From Source

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
mlx-gpt-oss --model mlx-community/gpt-oss-20b-MXFP4-Q8

API Endpoints

Chat Completions Notes

• model is required for compatibility, but the server always uses the single model loaded at startup.
• Supports OpenAI-style messages, stream, tools, tool_choice, stop, and common sampling params.
• top_k is accepted but generation remains pinned to top_k=0 for GPT-OSS behavior.
• reasoning_effort can be set directly, or via chat_template_kwargs.reasoning_effort.
• Streaming returns chat.completion.chunk events and ends with [DONE].

Responses API Notes

• Supported input types are text message items, replayed function_call items, and function_call_output items.
• Supported tools are custom function tools only.
• Stored responses are process-local, in-memory, and bounded by LRU eviction.
• previous_response_id reuses stored conversation transcript, but does not carry forward prior instructions.

Responses API Limits

• No multimodal inputs (image, audio, file, etc.)
• No hosted OpenAI tools such as web_search, file_search, or code_interpreter
• No structured output / non-plain-text text.format
• No parallel_tool_calls=false
• No named/required tool forcing; tool_choice supports auto and none

Tool Calling Reliability

• Uses official Harmony assistant-action stop tokens from openai-harmony (no hardcoded token IDs).
• Handles streaming edge cases: unfinished tool-call endings, buffered fallback dedupe, and repeated identical tool calls.
• Addresses a class of tool-calling failures seen in other MLX servers.

CLI Options

Security

• No built-in auth or API key checks, this is your responsibility.
• Default host is 0.0.0.0 for local/LAN self-hosting.
• CORS is permissive (*, credentials disabled).
• Use --host 127.0.0.1 for local-only access.