mlx-serve

Local inference server for Apple Silicon that hot-swaps MLX models on demand — text, vision, embeddings, TTS, STT, and decision (Laya) — loading exactly one at a time to stay within unified memory limits.

Client / LiteLLM  -->  mlx-serve (port 8095)  -->  MLX model (one at a time)

Install

pip install mlx-serve[all]

# or pick only what you need:
pip install mlx-serve[text,vision]
pip install mlx-serve[embeddings,tts,stt]
pip install mlx-serve[decision]       # Laya typed decision models

Requires: Apple Silicon Mac (M1+), macOS 13+, Python 3.11+

Quick Start

# 1. Generate a default config
mlx-serve init

# 2. Edit models.yaml to list your models (see docs/configuration.md)

# 3. Start the server
mlx-serve start

# 4. Verify
curl http://localhost:8095/v1/models

Why mlx-serve?

Key differences:

• vs Ollama — Ollama uses llama.cpp. mlx-serve uses Apple's native MLX framework, which typically achieves better throughput and memory efficiency on Apple Silicon. mlx-serve is what Ollama would be if it were built natively on MLX.
• vs LM Studio — Closed source, requires a GUI, cannot be embedded in headless pipelines.
• vs mlx-openai-server — Runs all models in-process, causing memory fragmentation over long sessions. mlx-serve isolates text/vision models as subprocesses so the OS reclaims all memory cleanly on unload.
• vs Docker — MLX requires direct Metal GPU access. Docker on Mac runs a Linux VM without Metal. The correct topology: stateless services in Docker, mlx-serve on the Mac host via host.docker.internal.

Features

• Hot-swap by model name — send a request to any configured model; the server loads it and unloads the previous one automatically
• OpenAI-compatible API — drop-in with LiteLLM, any OpenAI SDK, or direct HTTP
• All six MLX model types — text (mlx-lm), vision (mlx-vlm), embeddings (mlx-embeddings), TTS (mlx-audio), STT (mlx-whisper), decision (laya_mlx)
• Subprocess isolation — text/vision models run as isolated subprocesses; embeddings/TTS/STT run in-process
• Auto-unload on inactivity — configurable timeout (default 10 min) frees memory when idle
• Per-request keep_alive — override the idle timeout per request ("keep_alive": "30m", "-1" for permanent, 0 to unload immediately)
• Prompt caching — max_kv_cache_size per model caps KV cache token capacity for efficient prefix reuse
• Model management API — preload, force-unload, delete from disk, show detail, pull from HuggingFace
• Observability — request metrics (TTFT, TPS, latency), memory monitoring, lifecycle event log, dashboard endpoint
• Optional auth — set MLX_API_KEY to protect all /v1/* endpoints
• YAML config — add models by editing models.yaml, no code changes needed
• CLI — mlx-serve init, start, stop, status, logs

Supported Model Types

Usage

Chat completion

curl http://localhost:8095/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-qwen2.5-7b",
    "messages": [{"role": "user", "content": "What is Apple Silicon?"}]
  }'

Streaming

curl http://localhost:8095/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-qwen2.5-7b",
    "messages": [{"role": "user", "content": "Count to 5"}],
    "stream": true
  }'

Embeddings

curl http://localhost:8095/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "mlx-qwen3-embedding", "input": "Hello world"}'

Text-to-speech

curl http://localhost:8095/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "mlx-chatterbox", "input": "Hello from Apple Silicon."}' \
  --output speech.wav

Speech-to-text

curl http://localhost:8095/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=mlx-whisper-turbo"

Typed decisions (Laya)

curl http://localhost:8095/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "laya-en",
    "state": "I was billed twice. Please refund the duplicate.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this?",
        "criteria": ["billing", "technical", "sales"]
      },
      "urgent": {
        "type": "noul",
        "instructions": "Is this urgent?"
      }
    }
  }'

Returns typed answers with probabilities — no token-by-token decoding, no streaming.

LiteLLM Integration

mlx-serve is designed to sit behind LiteLLM in a Docker-on-Mac stack.

# litellm/config.yaml
model_list:
  - model_name: mlx-qwen2.5-7b
    litellm_params:
      model: openai/mlx-qwen2.5-7b
      api_base: http://host.docker.internal:8095/v1
      api_key: none

  - model_name: mlx-qwen3-embedding
    litellm_params:
      model: openai/mlx-qwen3-embedding
      api_base: http://host.docker.internal:8095/v1
      api_key: none

Development

git clone https://github.com/raspoli/mlx-serve.git
cd mlx-serve
make install    # uv sync with all extras
make dev        # start with auto-reload
make test       # run test suite
make lint       # ruff check + format check

See docs/development.md for the full guide.

Documentation

License

MIT