slim-serve

SLIM — Small Language Inference Machine. An OpenAI-compatible streaming server for local small language models (llama.cpp GGUF, MLX), with admission control, deadlines, cancellation and Prometheus metrics.

GitHub · Full README · Architecture · Benchmarks · Changelog

Install

pip install "slim-serve[llama]" \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu   # prebuilt CPU wheel

On Apple Silicon, pip install "slim-serve[mlx]" installs the MLX backend instead. Without the extra index, pip builds llama-cpp-python from source, which needs a C++ compiler.

Quick start

curl -L -o qwen.gguf \
  https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf
slim-serve serve --model qwen.gguf          # http://127.0.0.1:8000

No model? SLM_ENGINE=fake slim-serve serve serves deterministic fake tokens through the same stack.

The OpenAI Python SDK works against it as is:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
for chunk in client.completions.create(model="any", prompt="Hello", max_tokens=32, stream=True):
    print(chunk.choices[0].text, end="", flush=True)

What you get

• POST /v1/completions, streaming (SSE) and non-streaming, plus GET /v1/models.
• Inference on a dedicated thread per model replica, never on the event loop.
• A bounded queue that fails fast with 503 + Retry-After instead of piling up latency.
• One deadline per request; timeouts and client disconnects stop generation within a token.
• /healthz (liveness) and /readyz (model loaded), so orchestrators route correctly.
• Prometheus metrics that separate queue wait, time to first token and decode speed.

It doesn't batch sequences: for raw GPU throughput, use vLLM or llama-server. Why, and where it breaks down.

Configuration

SLM_* environment variables, a .env file, or serve flags (which take precedence):

All settings · slim-serve info prints the detected hardware and effective configuration.

Learn more

• Performance: measured queueing behaviour, serving overhead, prefill vs. KV cache, replicas
• Failure handling and metrics
• Design decisions
• SLOM: durable, tool-using agents built on top of slim-serve

MIT licensed.