mcp-ollama-vllm

An MCP server that lets an MCP client such as Claude Code talk to local language models directly. The backend is wired up at startup: Ollama, vLLM, or a generic OpenAI-compatible endpoint (set once via LOCAL_BACKEND, see below). The third, openai, is not a plugin loader for arbitrary tools; it is one more fixed backend implementation, covering servers such as LM Studio, the llama.cpp server, LocalAI, TGI, koboldcpp, SGLang or Jan, which all expose the same OpenAI-compatible /v1 endpoints as vLLM but are not vLLM itself.

Why this one

The one thing this server does that a plain Ollama/vLLM call does not: schema-validated JSON output, checked with its own dependency-free validator and automatically retried on a violation (up to 3 attempts total, feeding the concrete violations back to the model), the same way, with the same tool call, regardless of which of the two backends is configured. Both Ollama's format and vLLM's response_format steer the model toward JSON, but neither guarantees that the result actually matches your schema; this server checks that independently instead of trusting the backend's word for it.

Before / after (illustrative, not a real run):

// Backend's own schema enforcement alone (Ollama "format" / vLLM "response_format"):
// looks like valid JSON, but a required field went missing under load and nobody
// checked it against the schema before handing it onward.
{"amount": 42.10, "currency": "EUR"}

// local_structured: the same case, but validated against the schema and
// automatically retried with the concrete violation fed back to the model.
{"amount": 42.10, "currency": "EUR", "total": 42.10}
// -> attempts: 2, valid_on_first_try: false, outcome already re-validated

Known limits, stated on purpose:

• With vLLM the model listing reports less metadata than Ollama: no size or quantization, those fields read unknown/0 instead of a made-up value.
• With the generic openai backend the model listing usually reports even less: most such servers answer /v1/models with only id (and owned_by), so parameter_size/quantization read unknown, sizes read 0, there is no adapter signal, and max_context is only included when the endpoint genuinely reports it. The top-level metadata_reported field (see list_models below) makes this "not reported" honestly visible.
• The think parameter (reasoning trace) has no effect on vLLM or openai; it is only wired up for Ollama. See "The think parameter" below.

Operating systems: CI runs the self-tests on Linux, and the mock-backed setup self-test selftest_vllm.py on Windows too; the server is pure Python with no OS-specific calls, and its live backend path was additionally exercised once on Windows on 2026-08-07. Needs a running backend service (Ollama or vLLM); the setup self-test selftest_vllm.py uses a mock and needs no real service. macOS is not separately tested.

Purpose

A large part of an AI agent's work is pure language processing: writing, summarizing, classifying, extracting structured data from text. These parts can be offloaded to a local model, while searching, reading, writing and the orchestration stay with the calling client.

This is explicitly about a model call, not an agent. The local model receives text and returns text. It has no tools, no file access, and it does not work in multiple steps.

Security properties

These limits are part of the design, not just the docs:

• No file access. The server exposes no tool that reads or writes files. It reads only its own configuration from environment variables.
• No shell execution. There is no path to subprocess, os.system or anything comparable.
• Local machine or local network only. The server process itself makes no outbound calls other than to its configured backend endpoint, and that endpoint's address is checked both at startup and again on every call; if it points to the internet, the server refuses to start (or the call is rejected). Names are resolved for this, and all addresses behind them must be local, otherwise the guard could be bypassed via DNS; the resolved address is then pinned for the actual connection so a later re-resolution to a public address cannot slip past the check. Loopback and private networks (RFC 1918) are allowed, so a vLLM box on the local network is reachable; public addresses are rejected.
• Two direct dependencies: mcp and httpx. mcp in turn pulls in several further packages transitively (among others anyio, pydantic, starlette, uvicorn, jsonschema, pyjwt). The full resolved tree is pinned in uv.lock; CI checks with uv lock --check that the lock still matches pyproject.toml, then installs the exact locked tree for the tests, so a drifted lock fails CI. A separate, advisory pip-audit job checks the locked tree for known vulnerabilities on every push to main and pull request and again weekly; it does not block a pull request.

These properties bound what this server does; they are not a claim about the AI tool that drives it. The model you call runs locally, but the Claude/ChatGPT session that decides to call it is a separate hosted service and sees the calls it makes, like any MCP call. Drive the server with a local model if that matters.

See SECURITY.md for the trust boundaries this covers (the model as an oracle rather than an agent, prompt injection via model output, DNS rebinding pinning) and what it does not.

Installation

Replace /path/to/mcp-ollama-vllm below with the directory you extracted/cloned this project into.

Python 3.10 or newer is required. There are two equivalent ways to set up a virtual environment; use whichever tool you have.

With the standard venv module and pip:

cd /path/to/mcp-ollama-vllm
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

If python3 -m venv fails (on some distributions the python3-venv package is missing), either install that package or take the route with uv:

cd /path/to/mcp-ollama-vllm
uv venv .venv --seed
.venv/bin/pip install -r requirements.txt

(--seed makes sure pip lands in the environment. Alternatively, without --seed, install with uv pip install --python .venv/bin/python -r requirements.txt.)

The modules live under src/mcp_ollama_vllm/ (a src layout, so the package name does not collide with other MCP servers' top-level modules). Either of the above gives you a checkout that runs via PYTHONPATH=src (see "Entry for the MCP configuration" below), or install the package itself with .venv/bin/pip install ., which additionally provides the mcp-ollama-vllm console script and needs no PYTHONPATH.

A reachable Ollama or vLLM backend is a prerequisite for running:

• Ollama: install locally (see https://ollama.com), start it with ollama serve and pull at least one model (e.g. ollama pull llama3.2:3b); check with ollama list. For local_embed additionally pull an embedding model, e.g. ollama pull nomic-embed-text.
• vLLM: an OpenAI-compatible vLLM endpoint on this machine or one on your local network. The backend is switched only through the environment variables described below; the server itself does not change.

Entry for the MCP configuration

In ~/.claude.json, or the mcpServers section of your Claude Code config:

{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "ollama",
        "LOCAL_HOST": "http://localhost:11434",
        "LOCAL_TIMEOUT": "600"
      }
    }
  }
}

Replace /path/to/mcp-ollama-vllm with the absolute path you extracted this project into. Alternatively via CLI:

claude mcp add local-models --env PYTHONPATH=/path/to/mcp-ollama-vllm/src -- \
  /path/to/mcp-ollama-vllm/.venv/bin/python -m mcp_ollama_vllm.server

By default, claude mcp add binds the server to the current working directory; for global visibility across all directories, append --scope user.

Alternatively, after .venv/bin/pip install . (see "Installation" above), the PYTHONPATH entry is not needed and args can be ["-m", "mcp_ollama_vllm.server"] as is, or the installed console script can be used directly as command: /path/to/mcp-ollama-vllm/.venv/bin/mcp-ollama-vllm with no args.

First call

Restart your MCP client so it picks up the new server, then call list_models with no arguments. A response with your models proves the server starts, finds the backend and talks to it correctly, all in one step.

Environment variables

Variable	Default	Meaning
LOCAL_BACKEND	ollama	Backend: ollama, vllm or openai
LOCAL_HOST	per backend	Endpoint; must be local or on your own network
LOCAL_TIMEOUT	600	Read timeout in seconds
LOCAL_EMBED_MODEL	nomic-embed-text (Ollama)	Default embedding model
LOCAL_API_KEY	-	Optional; openai only. Sent as a Bearer header, never logged
LOCAL_EMBED_PATH	/v1/embeddings	Optional; openai only. Embedding endpoint path
LOCAL_SCHEMA_MODE	auto	Optional; openai only. auto, response_format or none (see local_structured below)

The default for LOCAL_HOST is http://localhost:11434 for Ollama and http://localhost:8000 for vLLM. For openai there is no default: the generic backend has no sensible port to guess, so LOCAL_HOST is required and the server refuses to start without it.

The old names OLLAMA_HOST, OLLAMA_TIMEOUT and OLLAMA_EMBED_MODEL are still read so existing configurations do not break. If both are set, the LOCAL_ name wins.

Switching to a remote vLLM endpoint

The vLLM box on the local network is addressed through only two variables; the server itself does not change:

{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "vllm",
        "LOCAL_HOST": "http://192.168.x.x:8000",
        "LOCAL_TIMEOUT": "600",
        "LOCAL_EMBED_MODEL": "BAAI/bge-m3"
      }
    }
  }
}

Both backends can also be registered in parallel, under different names (e.g. local-models and local-models-server), so both are available in the same session.

Registering the same server multiple times

There is nothing wrong with registering the same server under several names at once, each with its own LOCAL_BACKEND/LOCAL_HOST. That way different backends can be addressed in the same session, for example:

• one entry for a local Ollama (http://localhost:11434),
• another for a vLLM on a different machine on your network,
• a third for an Ollama on that same remote machine.

All three are then instances of one and the same server, just configured differently. Give them descriptive names and check before use which entry hits the intended machine and backend, rather than relying on the name alone.

The openai backend works the same way, no new routing mechanism: two generic-server entries on different ports, for example LM Studio on http://localhost:1234 and a llama.cpp server on http://localhost:8080:

{
  "mcpServers": {
    "local-models-lmstudio": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "openai",
        "LOCAL_HOST": "http://localhost:1234"
      }
    },
    "local-models-llamacpp": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "openai",
        "LOCAL_HOST": "http://localhost:8080"
      }
    }
  }
}

If the endpoint used for chat does not also serve embeddings (see local_embed below), register a second entry with a matching LOCAL_EMBED_PATH, or one pointing at a separate, embedding-capable backend entirely, rather than expecting one endpoint to do both.

The timeout is intentionally generous: local models can take minutes with a long context or under load. The connection setup has its own short limit of 10 seconds, independent of that, so a switched-off Ollama service surfaces immediately instead of hanging for ten minutes.

Other MCP clients

The server speaks plain stdio MCP, so any client that supports a command/args/env style server entry works the same way. The exact config file and key names differ per client; check its docs for where an mcpServers-style block (or equivalent) lives. Two generic examples:

// Checkout-based, PYTHONPATH points at src/:
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/python",
      "args": ["-m", "mcp_ollama_vllm.server"],
      "env": {
        "PYTHONPATH": "/path/to/mcp-ollama-vllm/src",
        "LOCAL_BACKEND": "ollama"
      }
    }
  }
}

// After `pip install .`, using the installed console script directly:
{
  "mcpServers": {
    "local-models": {
      "command": "/path/to/mcp-ollama-vllm/.venv/bin/mcp-ollama-vllm",
      "env": { "LOCAL_BACKEND": "ollama" }
    }
  }
}

Tools

list_models

No parameters. Returns per model the name, parameter size, quantization, on-disk size (GB and bytes), family and capabilities. The capabilities show, among other things, which models can do embedding.

With vLLM the backend does not report size and quantization; those fields then read unknown and 0 respectively, instead of inventing a value. In return, three fields are populated there:

• is_adapter: whether the entry is a LoRA adapter (vLLM exposes adapters as their own entries in the model list). Ollama's entries carry this field too, always False, since Ollama has no adapter concept.
• base_model: which base model the adapter belongs to (from the parent field). Ollama's entries carry this field too, always empty.
• max_context: the model's context length (max_model_len), where vLLM reports it. This one is exclusive to vLLM; Ollama's entries do not carry it at all.

At the top level there is additionally adapters as a count, backend as the identifier of the active backend, and metadata_reported (True/False): whether the backend actually reports the metadata fields above, so a caller can tell "not reported by this backend" (openai, generally False) from "reported, but happens to be empty".

local_ask

Parameter	Required	Default	Meaning
model	yes	-	Model name, e.g. llama3.2:3b
prompt	yes	-	The instruction
system	no	-	Role/behavior instruction
temperature	no	0	0 = as deterministic as possible
max_tokens	no	-	Cap on generated tokens
context	no	-	Text prepended to the prompt
think	no	false	Enable the model's reasoning trace (see below)

Return: answer, model, model_digest, backend, duration_seconds, tokens (input/output/total), tokens_per_second, finish_reason. Model name plus digest make a run traceable, even if the same label later points at different weights.

The think parameter

Thinking models (capability thinking in list_models, e.g. qwen3:4b) reason before the actual answer. This reasoning trace counts against the same token budget as the answer. With a tight max_tokens the answer is therefore empty, even though the call technically succeeded; measured were ~3000 to 4000 reasoning tokens per call.

That is why think defaults to false, and the bridge sends think: false with every Ollama call. Whoever explicitly wants the reasoning trace sets think: true.

The think field is deliberately always sent, even as false. Measured against Ollama 0.31.2: think: false is harmless on a model without the thinking capability, only think: true is rejected there with HTTP 400 "... does not support thinking". A preliminary capability lookup via /api/tags is thus unnecessary, and the error message in the single problem case names the cause clearly.

With vLLM the bridge accepts the parameter but does not act on it: the thinking control there is not yet wired up. The call form is kept identical for both backends on purpose. The generic openai backend accepts and ignores think the same way as vLLM, for the same reason: no generic reasoning control (reasoning_effort, chat_template_kwargs, ...) is in scope here.

local_structured

Parameter	Required	Meaning
model	yes	Model name
prompt	yes	Instruction on what to extract from which text
schema	yes	JSON schema of the desired result
system	no	Role/behavior instruction
think	no	Enable the model's reasoning trace, default false (see local_ask)

Passes the schema to the backend's schema enforcement and additionally validates the answer against the schema itself. With Ollama this goes through the format field, with vLLM through the OpenAI-compatible response_format with {"type": "json_schema", "json_schema": {...}} (as the current vLLM docs recommend). If an older vLLM build rejects that with HTTP 400, it falls back once to the older guided_json, so the bridge stays usable there too. With the generic openai backend the default is response_format with json_schema as well; if the server rejects that with HTTP 400, it falls back (unlike vLLM, there is no guided_json to try) to a schemaless call, whose result the built-in validator checks and, on a violation, retries with the concrete violations fed back, same as any other unvalidated answer. Which path was accepted is remembered process-wide, so later calls skip the doomed response_format roundtrip. LOCAL_SCHEMA_MODE overrides this detection: response_format always requires it (a rejection is then a hard error, not a fallback), none never sends it. On a violation, up to two retries, passing the model the concrete violations. Without that feedback a retry at temperature 0 would be word-for-word identical and thus pointless. Only then an error, but then with the violations and the invalid raw output.

Return: outcome (parsed object), attempts, valid_on_first_try, model, model_digest, backend, duration_seconds, tokens.

The validation covers the subset that format uses in practice: type (also lists of types), properties, required, additionalProperties, items (also tuple form), enum, const, minimum/maximum, minLength/maxLength, minItems/maxItems, anyOf/oneOf/allOf. Not a full draft: $ref, pattern and format are not validated. Any keyword not listed above is silently ignored rather than enforced, among others exclusiveMinimum/exclusiveMaximum, multipleOf, uniqueItems, patternProperties, if/then/else and contains.

local_embed

Parameter	Required	Default	Meaning
texts	yes	-	List of texts to embed
model	no	nomic-embed-text (Ollama)	Embedding model

Return: vectors, count, dimensions, model, backend, duration_seconds.

Self-tests

Two separate tests. selftest.py runs against the real Ollama; selftest_vllm.py is a fast offline mock gate that needs no backend at all. Beyond these two, the vLLM paths were in addition confirmed once against a real vLLM (see "What was measured against a real vLLM" below).

.venv/bin/python selftest.py --rounds 10 --model llama3.2:3b   # against real Ollama
.venv/bin/python selftest_vllm.py                              # against a mock

Both run without an MCP client. selftest.py calls the tools through the real FastMCP dispatch (mcp.call_tool), i.e. the same path a connected client takes, and pulls no models itself.

The Ollama test first checks the schema validator itself against a battery of known violations (the exact count grows as the validator's keyword coverage grows). Without this step a high schema hit rate would be worthless: it could also mean the validator simply flags nothing. Then the retry and abort logic is checked deterministically with replaced HTTP access, because a schema violation cannot be produced on command at the real model.

selftest_vllm.py starts its own tiny HTTP server that mimics the three OpenAI-compatible endpoints and shuts it down again afterwards. It checks both: that the answers are read correctly and that the sent requests are formed correctly (that response_format with json_schema is actually sent and not Ollama's format). It stays useful as a fast, offline, deterministic gate even now that the vLLM paths have also been confirmed against a real vLLM (see "What was measured against a real vLLM" below): the mock needs no backend and always runs the same. Your own vLLM build may still behave differently, so a check against it stays worthwhile.

Measured limits

The following numbers come from a test run on a single laptop with tight video memory (about 4 GB VRAM), Ollama with -np 1, i.e. a single processing slot. Read them as an order of magnitude, not a promise; on other hardware the runtimes come out differently.

What is fully green

Area	Result
Schema validator against known bad schemas	each correctly detected
Retry and abort logic (without a model)	8/8
list_models (Ollama)	8 models, fields complete
local_ask answer, tokens, time	correct; reported duration matches externally measured to within 0.04 s
local_ask context and max_tokens	both effective (max_tokens: 16 yielded exactly 16 output tokens)
local_embed	3 vectors, 768 dimensions, 2.05 s
meaning of the vectors	similar sentences 0.815, dissimilar 0.574
error paths (5 cases)	each an understandable message, no stacktrace
stdio handshake through a real MCP client	all four tools visible with correct required fields
network guard	LAN and loopback allowed, public host and public IP rejected
vLLM paths against the mock	70/70

Schema fidelity, the decisive metric

Collected with qwen3:4b over 30 different inputs, 10 per schema. Measured was how often the output was schema-conform on the first try, i.e. without the built-in retry:

Schema	n	conform on 1st try	failed	Duration s (min/median/max)
A flat (enum, numeric bounds, additionalProperties: false)	10	10/10	0	35 / 56 / 516
B nested object (2 levels)	10	10/10	0	56 / 93 / 832
C list of objects (minItems, enum per element)	10	10/10	0	63 / 107 / 827
Total	30	30/30 (100 %)	0	35 / 93 / 832

Every delivered object additionally passed the independent re-validation against its schema, and the contents were factually correct (example B: sender, net amount, currency and reminder flag extracted correctly from prose).

How to read these 100 %. The backend's grammar binding already forces the JSON form at generation time; the rate therefore mainly measures whether the schemata are tight enough, not whether the model is clever. It holds for these three schemata, for this model, and for short German texts. It weakens as expected for schemata full of free strings without enum and without bounds. The retry loop was not needed a single time in this series, and it was not triggered in the real vLLM run either (local_structured came back valid on the first try there too, attempts: 1); its correctness is instead proven deterministically with replaced HTTP access.

On the runtime, and what skewed it

The spread from 35 s to 832 s does not come from the server. During the first half of the measurement a second, unrelated test ran on the same machine (at times a Python process at 1011 % CPU, load average 21 to 28). Ollama runs here with -np 1, i.e. a single processing slot for the whole machine. Measured consequences:

• Under foreign load a call took 309 to 832 s, without it 35 to 107 s.
• One call ran into the default timeout of 600 s and failed.
• A switch to llama3.2:3b was impossible at all under foreign load: as long as another model holds the slot, the request for a not-yet-loaded model waits indefinitely. Two attempts aborted after 400 s with no answer.
• The obvious way out, num_gpu: 0 (compute purely on the CPU to sidestep the 4-GB-VRAM contention), did not help, because the bottleneck was the CPU, not the GPU.

None of this is a fault of this server, but it is the practical limit of the approach on a single machine, and that is why it is here.

What was measured against a real vLLM

The vLLM paths are backed against a mock server: correct endpoints, correctly formed requests (response_format with json_schema), fallback to guided_json, LoRA detection, unified metadata, understandable errors. Beyond that, the tools were run once against a real vLLM: an OpenAI-compatible endpoint serving Qwen2.5-1.5B-Instruct (max_model_len 8192), driven through the real FastMCP dispatch (mcp.call_tool), the same path a connected client takes.

• list_models -> GET /v1/models: backend reported as vllm, the model recognized, max_context 8192, adapters 0.
• local_ask -> POST /v1/chat/completions: answer text, token counts (input/output/total) and tokens_per_second, finish_reason stop.
• local_structured -> POST /v1/chat/completions with response_format json_schema: the schema-conform object back, valid on the first try (valid_on_first_try: true, attempts: 1). This is exactly the schema-enforcement path, now confirmed against a real vLLM and not only the mock.
• local_embed -> POST /v1/embeddings: 404 from the backend, which the server caught cleanly as a clear message ("backend reports: Not Found ...; with vLLM the model is fixed when the service starts"). This is correct behavior, not a fault: the test vLLM served a chat model, not an embedding model, so there was no embedding endpoint to hit, and the server said so plainly.

What genuinely stays open, therefore, is the vLLM embedding path: it was measured against a real Ollama, but not against a vLLM deployment that serves an embedding model (the test vLLM was chat-only). Whoever uses vLLM embeddings must run an embedding-capable vLLM; against a chat-only vLLM the server deliberately returns a clear error. Beyond that, your own build may still differ in whether it takes response_format or the legacy guided_json path, and in schema fidelity with a larger model and a LoRA adapter.

What follows for operation

• One model at a time. With -np 1 the local Ollama instance is a resource two parallel agents cannot share. Before a longer run, a glance at curl localhost:11434/api/ps pays off.
• The 600 s timeout is tight, not generous. Under foreign load it was exceeded. For measurement runs set LOCAL_TIMEOUT higher.
• Thinking models stay more expensive, but are no longer dangerous. qwen3:4b produced 186 output tokens for the answer "Paris", because it reasons before answering. For classifying and extracting, a model without a thinking step (llama3.2:3b, granite4:micro) remains faster and cheaper. For qwen3 variants the think parameter (default false) controls the reasoning trace: the bridge switches it off by itself, and a tight max_tokens then yields an answer instead of an empty string.
• Build long runs to be resumable. Otherwise every abort under foreign load destroys all values collected so far.

Pitfalls

• Ollama's format does not guarantee a valid result. The grammar binding forces the JSON form, but not reliably required, enum or numeric bounds. That is exactly why this server additionally validates itself. Whoever uses format without their own validation gets silently incomplete objects.
• Small models need a tight schema. The more enum, required and bounds are set, the more usable the result. A schema of nothing but free strings invites the model to make things up.
• VRAM contention is the biggest time sink. If two requested models do not fit into video memory at once, Ollama reloads on every switch. A call then takes minutes instead of seconds, without anything being wrong at the server. Work with a single model where possible.

License

MIT, see LICENSE.