Modelship

Your agents already speak OpenAI. Give them a /v1/responses backend you own.

Modelship serves the whole agentic surface — reasoning, tool calling, server-side conversation state, and MCP servers the gateway calls for you — and scores 17/17 on the independent Open Responses conformance suite. Embeddings, speech, and image generation ride the same /v1. Change the base URL and your agent runs unchanged, on hardware you control.

Quick Start

One command, one model. Pick your hardware:

CPU — Linux, Windows, or macOS via Docker (images are multi-arch: amd64 + arm64):

docker run --rm --shm-size=8g -p 8000:8000 -v modelship-cache:/.cache \
  ghcr.io/modelship-ai/modelship:latest-cpu start \
  --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-cpus 4

NVIDIA GPU — needs the NVIDIA Container Toolkit:

docker run --rm --shm-size=8g --gpus all -p 8000:8000 -v modelship-cache:/.cache \
  ghcr.io/modelship-ai/modelship:latest-cuda start \
  --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-gpus 1

Apple Silicon — native, with Metal offload:

uv tool install mship && mship bootstrap --metal
mship start --model "Qwen/Qwen3-8B-GGUF:*Q4_K_M.gguf" --loader llama_server \
  --usecase generate --num-gpus 1

Wait for the Deploy … succeeded log line, then talk to it — this hits the Responses API and streams the model's reasoning as it thinks:

uvx --with httpx llm openai endpoint http://localhost:8000/modelship/v1 \
  -m qwen3-8b --responses "Which is larger, 9.11 or 9.9?"

Add --chat for an interactive session, -T to hand it a tool, or --models to list what's deployed.

The model serves as qwen3-8b, inferred from the reference. It pulls ~5 GB and wants ~10 GB of free RAM; --num-cpus 4 reserves four cores for it, so lower it if the container has fewer (the deploy waits for resources it can't get). On a small box, swap in lmstudio-community/Qwen3-0.6B-GGUF:*Q4_K_M.gguf.

Deploying several models at once uses a models.yaml instead; one model's tuning blocks have flags too (--llama-server-config.n-ctx 8192) — see Model Configuration. Hitting an error? Check Troubleshooting.

curl http://localhost:8000/modelship/v1/responses \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8b", "input": "Which is larger, 9.11 or 9.9?"}'

Open Responses conformance

/v1/responses is tested against the independent Open Responses compliance suite (bun run test:compliance), which drives the endpoint over real HTTP against a live deployment rather than mocks.

Latest result: 17/17 — every core, compaction, vision, and WebSocket test (Qwen3-VL-8B-Instruct AWQ 4-bit, vLLM, 2026-10-04).

Loader parity: the vllm and llama_server loaders return the same result on all 17 tests.

Fourteen cluster guarantees — zero-downtime model cutover, load-driven autoscaling, engine crash recovery, fractional-GPU multi-tenancy, server-side MCP tool loops — are likewise asserted end-to-end against a live cluster with real models over real HTTP. See Verified cluster behaviour.

Why Modelship

• Conversations that survive the replica that started them. previous_response_id, reasoning, and tool state live in one store every gateway replica shares — a Ray actor by default, Redis when you want it to outlive the cluster. Scale the gateway out without sharding your users onto specific replicas.
• One endpoint for the whole app. Chat, embeddings for RAG, speech-to-text, text-to-speech, and image generation on a single OpenAI-compatible /v1 — instead of a service per modality.
• A stack that fits the hardware you have. Allocate GPU fractions per model (70% for the LLM, 5% for TTS), or run the same model CPU-only. vLLM, llama.cpp, Diffusers, sherpa-onnx, and whisper.cpp coexist in one deployment.
• Changes you can ship on a Tuesday. Edit the model set and reconcile: models are added, replaced, or dropped incrementally, with a blue-green cutover per model and no gateway restart.

Supported OpenAI endpoints

Architecture

Each model runs as an isolated Ray Serve deployment with its own lifecycle, health checks, and resource budget. Several inference backends are available:

A model name maps to exactly one deployment — swapping GPU/CPU or backend replaces it with a blue-green cutover rather than adding a second one alongside it. Each deployment scales horizontally with num_replicas, load-balanced by Ray Serve. Multiple gateways can share one cluster and one port (--gateway-name), each mounted at /<name>/v1 with its own model set.

More ways to run

Several models at once — a models.yaml replaces the flags and unlocks the nested tuning blocks (vllm_engine_kwargs, llama_server_config, autoscaling_config):

docker run --rm --shm-size=8g -p 8000:8000 -v modelship-cache:/.cache \
  -v ./models.yaml:/models.yaml \
  ghcr.io/modelship-ai/modelship:latest-cpu start --config /models.yaml

See Model Configuration for the full reference, and config/examples/ for working files per backend.

Native install, any node role — one install command everywhere; pick the role once, at bootstrap:

uv tool install mship      # or: pipx install mship / pip install mship

mship bootstrap --cuda     # NVIDIA GPU node
mship bootstrap --cpu      # CPU node (includes vLLM CPU)
mship bootstrap --metal    # Apple Silicon
mship bootstrap --thin     # coordinator/head only, no capacity

mship start --config models.yaml

mship bootstrap installs a pinned Python 3.12.10 environment for that role and records it, so deploy afterwards needs no variant flag and installs nothing. A new node joins the cluster by running the same two commands. Platform prerequisites apply — see Native install.

High-throughput GPU serving — use --loader vllm with safetensors or AWQ/GPTQ/FP8 weights instead of GGUF, and set HF_TOKEN for gated models. Multi-node — join hosts into one Ray cluster with plain docker run, or deploy on Kubernetes with the Helm chart.

Documentation

Full docs are hosted at docs.model-ship.ai. The same source files are browsable in this repo:

• Installation — requirements, node roles, and the Docker / native / Helm guides
• Model Configuration — full models.yaml reference, GPU pinning, environment variables
• Architecture — system design, request lifecycle, loaders
• Production Readiness — verified cluster behaviour and the hardening roadmap
• Monitoring & Logging — Prometheus metrics on port 8079, Grafana dashboard, structured logging, health checks
• Multi-node without Kubernetes — join VMs into one Ray cluster with plain docker run
• Development — dev environment setup, building, and running locally
• Troubleshooting — common first-run errors and fixes

Contributing

See CONTRIBUTING.md for guidelines on setting up the dev environment, code style, and submitting pull requests.