ServeLLM

Expose a local LLM runtime (Ollama, vLLM, or llama.cpp) as a reliable, OpenAI-compatible HTTP/WebSocket API.

Install

pip install servellm

Quickstart

servellm init      # writes servellm.yaml with a generated API key
servellm start      # starts the server in the background
servellm models      # list available models

Usage

OpenAI-compatible endpoint, so existing OpenAI clients work by changing base_url:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast", "messages": [{"role": "user", "content": "hello"}]}'

Also available: GET /v1/models, GET /health, GET /metrics, WS /ws/chat, and the simpler POST /chat endpoint.

Configuration

Edit servellm.yaml (created by servellm init) to point at your runtime, set API keys, concurrency limits, and model aliases. No Python code required. See the project repository for the full configuration reference.

CLI

servellm init      create servellm.yaml in the current directory
servellm start      start the server
servellm stop      gracefully stop the server
servellm status      check whether the server is running and healthy
servellm models      list configured/discovered models

License

MIT