llm-router-gate

Installation

Install globally to use as a CLI

uv tool install llm-router-gate

Or install within your project to use as an SDK

uv add llm-router-gate

Ephemeral call

You can also invoke the CLI from PyPI without any local installation:

uv x llm-router-gate --help

What it is for

llm-router-gate is a single-shot, single-turn LLM probe with the HTTP layer left visible. It sends exactly one user message to exactly one model, streams the answer, and then prints everything a developer normally has to reach for a proxy or curl -v to see:

• the outgoing request line, all headers, and the pretty-printed JSON body;
• the response status line and all response headers (rate-limit counters, request-id, cf-ray, content-type: text/event-stream, …);
• the raw SSE lines as they arrive (RAW STREAM: data: {...});
• a reconstructed final response payload, with generated text truncated so the structure stays readable;
• a token/cost report, with the provenance of every number it prints.

It is deliberately not a chat client. There is no conversation history, no system prompt, no tool use, no retries. Think of it as the LLM equivalent of curl -v plus a calculator: use it to answer "what exactly goes over the wire, what exactly comes back, and what did that cost?"

Good uses:

• comparing how three backends (Anthropic, llama.cpp, OpenRouter) frame the same prompt and the same concepts (reasoning, usage, finish reasons);
• checking that a local llama.cpp build honours a body field you just added;
• getting a hard cost number for a prompt before wiring it into a loop;
• debugging "the model returned nothing" — the raw stream dump usually shows the text arrived on a channel your client wasn't reading.

Requirements

The project uses uv; run everything through it so the lockfile is respected:

uv run src/router.py "..." [options]

Credentials are read from the environment, with .env loaded automatically via python-dotenv (load_dotenv() at import time):

Provider	Needs
anthropic	ANTHROPIC_API_KEY (read by the anthropic SDK)
openrouter	OPENROUTER_API_KEY
local	nothing — a llama.cpp server on port 8090

Note that the request logger prints every header, including x-api-key and Authorization. That is the point of the tool, but it means: do not paste raw output into an issue, a PR, or a chat window without redacting those two lines first. (The transcript above has them replaced with XXXXX by hand.)

CLI reference

After installing with uv tool install, you may run the command directly from any terminal window:

llm-router-gate PROMPT [-p {anthropic,local,openrouter}] [-m MODEL]
                         [-t MAX_TOKENS] [--local-url URL]
                         [--reasoning | --no-reasoning] [-y]

Flag	Default	Meaning
PROMPT (positional)	—	Sent verbatim as a single {"role": "user"} message. Quote it.
-p, --provider	anthropic	Which backend to hit. Selects the code path and the cost behaviour.
-m, --model	per-provider	Required for anthropic (argparse errors out otherwise). Defaults to ggml-org/SmolLM3-3B-GGUF:Q4_K_M for local, z-ai/glm-5.2:free for openrouter.
-t, --max-tokens	1024	Output cap. For Anthropic it is also the worst case in the pre-flight cost estimate.
--local-url	http://localhost:8090/v1/chat/completions	Point at a different llama.cpp / vLLM / LM Studio instance.
--reasoning / --no-reasoning	omitted → provider default	Tri-state thinking toggle. See below.
-y, --yes	off	Skip the Proceed with API call? (y/N) gate. Use in scripts; keep it off when experimenting against a paid model.

Examples

# Paid call, with a confirmation gate and a real cost report.
llm-router-gate "Explain entropy" -m claude-haiku-4-5 -t 500

# Local llama.cpp, defaults to SmolLM3-3B.
llm-router-gate "Explain entropy" -p local -t 200 -y

# Same, thinking forced off.
llm-router-gate "Say hello" -p local -t 100 -y --no-reasoning

# OpenRouter, thinking forced on.
llm-router-gate "Explain entropy" -p openrouter -m z-ai/glm-5.2:free --reasoning

# A different local server.
llm-router-gate "Hi" -p local --local-url http://localhost:1234/v1/chat/completions -y

Two code paths, on purpose

main() dispatches to one of two coroutines. They are kept separate because the two wire protocols disagree about almost everything except "it's a POST".

run_anthropic() — the SDK path

Uses anthropic.AsyncAnthropic with an injected httpx.AsyncClient carrying event_hooks. That injection is the whole trick: the SDK does the auth, framing, SSE parsing and retry logic, while the hooks still let you see the raw traffic. It's the pattern to copy whenever you need observability without reimplementing a vendor SDK.

Sequence:

1. GET /v1/models — resolves the model's display_name and proves the key works.
2. POST /v1/messages/count_tokens — the exact input token count, including framing overhead. Falls back to the ~4 chars/token heuristic if it fails.
3. Prints a pessimistic estimate: exact_input × in_rate + max_tokens × out_rate, i.e. the cost if the model uses its entire budget. Then the y/N gate.
4. Streams via client.messages.stream() / stream.text_stream.
5. get_final_message() → dumps the payload, then reports actual usage and cost from usage.input_tokens / usage.output_tokens.

Pricing comes from FAMILY_PRICING_TIERS, matched by substring on the lowercased model id (opus / sonnet / haiku), defaulting to Sonnet rates. These are hardcoded base rates in $/1M tokens and are the part of this script most likely to be stale — they ignore long-context surcharges, batch discounts, and cache read/write rates. Treat the cost figure as an order of magnitude, and update FAMILY_PRICING_TIERS when Anthropic's price list moves.

Extended thinking is not wired on this path: --reasoning is silently ignored for -p anthropic because no thinking block is ever sent.

run_openai_compatible() — the hand-rolled path

One httpx.AsyncClient.stream("POST", ...) and a hand-written SSE reader, shared by local and openrouter. Differences between the two are gated on is_local:

	local (llama.cpp)	openrouter
Auth	none	Authorization: Bearer $OPENROUTER_API_KEY
Usage in stream	not requested	stream_options.include_usage: true
Reasoning toggle field	chat_template_kwargs.enable_thinking	reasoning: {"enabled": bool}
Token counts	timings.predicted_n / prompt_n on last chunk	usage.completion_tokens / prompt_tokens

The reader loop: skip anything not prefixed data:, break on [DONE], json.loads the rest, keep the last chunk as final_chunk, and latch chunk["usage"] whenever it appears.

The reasoning/answer two-channel model

This is the subtlety that motivated most of the current code, and the thing to understand before extending it.

An OpenAI-compatible delta can carry generated text on more than one field, and different servers name them differently:

delta.reasoning_content  ──→ reasoning accumulator   (llama.cpp)
delta.reasoning          ──→ reasoning accumulator   (OpenRouter)
delta.content            ──→ answer accumulator      (everyone)
choice.text              ──→ answer accumulator      (legacy completions-style)

They are accumulated into two separate lists and rendered differently: reasoning is printed dimmed, the answer is printed at normal brightness. The final payload reflects both — message.content is the answer, and a message.reasoning key is added only if a reasoning trace was actually received.

Two consequences worth internalising:

1. A client that reads only delta.content will report an empty response from a thinking model, even though the server streamed hundreds of tokens. That is not a server bug and not an empty generation; it is a channel mismatch.
2. Reasoning tokens are charged against max_tokens. A thinking model given -t 10 can burn the entire budget mid-thought and terminate with finish_reason: "length" before emitting a single answer token. If you get an empty answer from -p local, raise -t (try 100+) or pass --no-reasoning before suspecting anything else.

--reasoning semantics

Implemented with argparse.BooleanOptionalAction and default=None, giving three distinct states rather than the usual two:

Invocation	args.reasoning	Behaviour
omitted	None	No field is sent. The provider/model default stands.
--reasoning	True	Force thinking on.
--no-reasoning	False	Force thinking off.

The tri-state matters: a plain store_true flag cannot express "don't express an opinion", and silently sending false is not the same request as sending nothing. The active state is echoed in the pre-request panel as Reasoning: provider default | on | off.

For local, the flag becomes chat_template_kwargs: {"enable_thinking": ...}. llama.cpp forwards chat_template_kwargs into the Jinja chat template, and SmolLM3's template reads enable_thinking. This is therefore a template-level knob: against a GGUF whose template does not reference that variable, the field is accepted and quietly does nothing. If the toggle appears to have no effect, inspect the model's chat template before blaming the client.

Token reporting and its provenance

Non-Anthropic providers get no cost figure at all — only counts, and every panel names its Source: so you never have to guess how solid a number is. Resolution order in run_openai_compatible():

1. provider usage block — usage.prompt_tokens / usage.completion_tokens from the stream. Authoritative. This is the OpenRouter path.
2. llama.cpp timings block — final_chunk["timings"]["predicted_n"] and prompt_n. Authoritative, and the normal local path.
3. heuristic ~4 chars/token — len(text) // 4. Last resort only.

Output tokens are labelled "Output Tokens (generated, reasoning included)" because predicted_n counts everything the model produced, reasoning and answer alike. Never derive this number from len(full_content): with a thinking model that undercounts massively — an answer-less run would report 1 output token against a real predicted_n of 10.

The input count in the pre-request panel is always the crude heuristic for these providers (there is no local count_tokens endpoint); the post-request panel replaces it with a real number when the server supplied one.

Extending it

• Truncation. RESPONSE_TEXT_PREVIEW_CHARS = 50 controls how much generated text survives into the final-payload dump. print_final_payload deep-copies via json.dumps/loads before truncating, so the live objects are untouched; it handles both the OpenAI shape (choices[0].message.content / .reasoning) and the Anthropic shape (content[] blocks with a text key). Full text is still streamed to stdout — only the JSON echo is shortened.
• The RAW STREAM: lines are unconditional in the read loop. They are the most useful thing in the output when diagnosing a new server, and the most noisy otherwise. Gate them behind a --raw flag if that becomes annoying.
• Error handling is intentionally flat. Both paths catch broad Exception, print it red, and return; there are no retries and no non-zero exit on API failure. Fine for a probe, wrong for anything automated — add explicit exit codes before putting this in a pipeline.
• A new OpenAI-compatible backend usually needs only a URL, an auth header and possibly a new reasoning field name. Add it to the is_local branches rather than forking the reader; the dual-channel accumulator already covers every delta shape seen so far.
• Wiring Anthropic extended thinking means adding a thinking={"type": "enabled", "budget_tokens": N} argument in run_anthropic() and consuming thinking content blocks from the stream, so --reasoning stops being a no-op there. Currently unimplemented.