lmrelay 0.0.5

lmrelay - a credentialed relay beside a local Ollama

If you work with Ollama, you run into this: by default it is reachable only from localhost, and it has no built-in authentication. Connecting to Ollama from another machine usually means changing its systemd configuration, or putting a reverse proxy in front of it. lmrelay solves that. It installs with pip and runs as a daemon beside Ollama: it listens on a port of its own and, when you want it to, requires a credential for access.

English | Español | Português | Français | Deutsch | Italiano | Русский | 中文 | 日本語 | हिन्दी | 한국어

Requirements

• Python 3.11 or higher, and four dependencies: FastAPI, starlette, uvicorn and httpx.
• Linux and macOS run every command, including serve (detached) and enable: a systemd --user unit on Linux, a launchd agent on macOS, and a refusal where neither is installed.
• Windows runs run only. serve reports that the platform has no os.fork, and enable that there is no systemd or launchd, rather than half-starting.
• A local Ollama on 11434 is the default upstream, but it is not required. A relay with only hosted providers configured is valid, as long as default_upstream names one of them.

Installation

pip install lmrelay

Or the current main, which may be ahead of the release:

pip install git+https://github.com/wachawo/lmrelay.git

Quick start

lmrelay init     # writes ~/.lmrelay/lmrelay.toml
lmrelay run      # foreground, port 11435

Ollama keeps 11434 and its installation is left exactly as it is. Clients are repointed at 11435 instead. That is the trade: nothing about an existing Ollama has to change, and the relay is opt-in per client.

Auth is off in a fresh state, so on loopback this is a transparent proxy in front of Ollama. That is deliberate: a relay you have just installed should not lock you out of your own Ollama before you have a token. Point a client at 11435 and it works:

OLLAMA_HOST=127.0.0.1:11435 ollama list

Checking it works

Ask the relay for the model list. Either dialect will do; both reach the same Ollama:

curl http://127.0.0.1:11435/api/tags    # Ollama's shape
curl http://127.0.0.1:11435/v1/models   # OpenAI's shape

Then put a model to work. qwen3:8b here is whatever ollama list shows on your machine:

curl http://127.0.0.1:11435/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Reply with exactly: it works",
  "stream": false,
  "think": false
}'

curl http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "say ok"}]
}'

qwen3 reasons before it answers, and only Ollama's dialect has a switch for that: the "think": false above. Through /v1/chat/completions the reasoning arrives inside the content as a <think> block, because lmrelay forwards what the upstream produced and does not edit it.

With auth on, every one of these needs the credential:

curl http://127.0.0.1:11435/api/tags \
  -H "Authorization: Bearer $TOKEN"

Running it for real

lmrelay token gen --label laptop   # printed once, never again
lmrelay auth true                  # now start requiring it
lmrelay enable                     # start at login, and start now
lmrelay status

lmrelay      running (pid 40213), healthy
listening    127.0.0.1:11435
config       /home/u/.lmrelay/lmrelay.toml
state        /home/u/.lmrelay/state.json
upstreams    anthropic, ollama, openai (default: ollama)
auth         on, 2 tokens
limits       total 10 per 30m, 10 at once
autostart    systemd: enabled, active

enable registers a systemd --user unit on Linux or a launchd agent on macOS, then starts it. From then on stop, restart and reload go through that manager instead of the pidfile, so the two cannot disagree about who owns the process. On a POSIX box with neither manager, lmrelay serve runs the relay detached.

Limiting what a caller may ask for

lmrelay limits set total 1              # one request at a time
lmrelay limits set total 1 60s          # one a minute, and still one at a time
lmrelay limits set per_address 10 30m   # ten every half hour
lmrelay limits set per_token 0          # off

Three scopes, one number each. requests is how many a caller may have in flight at once; add a period and the same number is also how many they may start in that long. A request must pass every scope you set.

If you set one number, set total. It is the one that protects the machine: ten callers each inside their own limit still arrive together, and a per-caller cap cannot see that. per_token beside it is what keeps one client with fifty threads from owning all of it.

A refused caller gets a 429 naming the scope, and a Retry-After when the relay can work one out honestly:

lmrelay: the relay's rate limit is exceeded: 10 per 30m ([limits.total])

The command writes into lmrelay.toml and leaves the rest of the file alone, comments included, then signals a running relay.

Usage

run, serve and restart take --host and --port. provider add takes --base-url, --dialect and a repeatable --header K=V; with a known name (openai, anthropic, deepseek, grok, ollama) the base URL, dialect and header shape come from a preset, so lmrelay provider add openai sk-... is the whole command. export takes --no-secrets, both it and import take --force to write over what is already there, and with no path at all the bundle goes to stdout and is read from stdin, so lmrelay export | ssh other-host lmrelay import moves a relay in one line. --config PATH is accepted by every command that reads the config or the state, which is every command except init, which always writes ~/.lmrelay/lmrelay.toml, and disable, which reads neither.

Choosing an upstream

The first path segment selects the upstream if and only if it exactly matches a key in [upstream]. Otherwise default_upstream handles the request and the path is untouched.

POST /api/chat                     -> ollama     /api/chat
POST /v1/chat/completions          -> ollama     /v1/chat/completions
POST /openai/v1/chat/completions   -> openai     /v1/chat/completions
POST /anthropic/v1/messages        -> anthropic  /v1/messages
POST /deepseek/v1/chat/completions -> deepseek   /v1/chat/completions
POST /grok/v1/chat/completions     -> grok       /v1/chat/completions

So a client only has to learn the port once, and retargeting one at a different provider is a single line:

from openai import OpenAI
from anthropic import Anthropic

OpenAI(base_url="http://relay:11435/openai/v1", api_key=RELAY_TOKEN)
OpenAI(base_url="http://relay:11435/v1", api_key=RELAY_TOKEN)  # Ollama
Anthropic(base_url="http://relay:11435/anthropic", api_key=RELAY_TOKEN)

curl http://127.0.0.1:11435/api/chat \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "model": "llama3",
  "messages": [{"role": "user", "content": "hi"}]
}'

GET /healthz answers {"status": "ok"} without touching an upstream and without a credential. GET /metrics answers a Prometheus scrape of aggregate counters and does need one, because it says how the relay is used rather than only that it is alive. Everything else goes through the relay.

Compatibility

lmrelay forwards the method, path, query string and body bytes unchanged, and it does not translate between API dialects.

¹ Ollama serves an OpenAI-compatible surface at /v1/* alongside its native /api/*. This is the practically important cell: an OpenAI-shaped client reaches all of ollama, openai, deepseek and grok by changing only the path prefix.

The four cases that do not work, and the reason each one cannot be made to work, are in the configuration document.

Where lmrelay can tell that a path certainly does not exist upstream, it says so itself rather than letting the provider's 404 look like your mistake:

lmrelay: upstream 'anthropic' speaks the Anthropic API;
'/v1/chat/completions' is an OpenAI-dialect path. lmrelay
forwards requests unchanged and does not translate between
dialects.

Every error lmrelay generates begins with lmrelay: , so it is never mistaken for something the provider said.

Configuration and Errors - the config file, caller tokens, providers, autostart, streaming behaviour, and what every error means.

Testing

pip install -e '.[test]'
pytest

Most of the suite drives the app in process against a recording upstream, so it needs no network and no Ollama. tests/test_streaming.py is the exception: it runs the relay under uvicorn in front of an upstream that answers a chunk at a time, because the property it checks, that the caller has the first line before the upstream has written the last, cannot be seen through an in-process client.

Why not nginx?

nginx already reverse-proxies, so a daemon has to earn its place. Briefly, point by point:

• Provider keys end up inside nginx.conf. A location and a proxy_set_header Authorization "Bearer sk-..." for each one, plus proxy_ssl_server_name on when the upstream speaks TLS. Here it is one command, and the key lives in a 0600 file rather than in a root-owned 0644 one.
• Checking a caller's token in nginx puts the tokens in nginx.conf too. A map and an internal location do it without a backend, but each token becomes a plaintext line in that same root-owned file, and adding or revoking one takes an edit and a reload.
• htpasswd has no ids or rotation. lmrelay token gen --label laptop, token list and token delete 1 do.
• nginx's defaults break streaming. proxy_buffering is on and proxy_read_timeout is 60s, and a large local model can think for longer than a minute before its first token. Both have to be found and turned off, usually after an answer has been cut in half.
• A wrong-dialect path gets the provider's own 404 through nginx. For the shapes it recognises, such as an Anthropic path sent to an OpenAI upstream, the relay answers 400 in its own words, so the mistake is not misread as the provider's.
• nginx ships with neither macOS nor Windows. pip install works the same on both.
• An SDK cannot be pointed at auth_basic the documented way. It accepts Basic and refuses everything else, while every SDK puts its key in Authorization: Bearer. Credentials in the URL do get through, but then api_key is dead weight: httpx writes the URL's credentials into that same header and the bearer never leaves. Every example in the provider's own documentation has to be rewritten.

Where nginx wins: TLS, already being installed, and rate limiting that holds up outside one process. The first two are not coming. lmrelay does have limits in three scopes, per credential, per address and for the relay as a whole, and the first of those is keyed on the caller's token in a way nginx cannot manage without holding the tokens itself. They are counted in this one process. The two compose rather than compete. Put nginx in front for TLS, and leave tokens, providers and limits here.

License

MIT License. See LICENSE.