mellontoken

Client for the mellontoken gateway.

import mellontoken

answer = await mellontoken.ask("Summarise this tender in one line.")
parsed = await mellontoken.ask(document_text, schema=DocExtraction)
full   = await mellontoken.ask_full(document_text, schema=DocExtraction)
full.cost_usd     # "0.00059550"
full.model        # the model that actually served it

Letting the gateway pick the model

answer = await mellontoken.ask(prompt, model="auto")
answer = await mellontoken.ask(prompt, model="auto", tier="high")   # bound the spend

full = await mellontoken.ask_full(prompt, model="auto")
full.route.model_slug    # what it picked
full.route.task          # the grid row it read
full.route.explanation   # why, in a paragraph you can argue with

One command, and one request. model="auto" goes to the gateway as the model name; the gateway classifies the prompt on three axes (task, effort, capability), reads the resulting grid cell, and serves the answer from whatever that cell names — all inside the same call. ask_full reports the decision on Answer.route, so routing is legible after the fact without a second call to ask about it.

Two of those three axes are read locally and cost nothing. A distilled model on the gateway reads the task and the effort in one forward pass. Only the capability gate goes to a provider — the distilled model has no capability head, and an unread gate routes "draw me a diagram of this schema" to a model that cannot draw, then bills for the failure. So a default ask is one classifier call, around $0.0002. llm=True puts all three axes on the LLM classifier instead: three calls, roughly $0.005.

Either way they land on the ledger as kind=classifier rows, so the spend is visible rather than hidden. A service sending the same shape of prompt a thousand times should read full.route.model_slug once and then name that slug: the decision is a property of the workload, not of each call.

tier= bounds what the router may spend and outranks the effort classifier, because a tier is a statement about your budget rather than a property of the text. Set it per call, or once via MELLONTOKEN_TIER. It is only read on the auto path — with a named model it is logged as ignored rather than silently dropped.

A router that resolves nothing is not an error. When the capability gate rejects every candidate in the cell the gateway answers 422 with the reason; ask logs it and retries against MELLONTOKEN_MODEL, which is what that error advises — keep your own model, so auto is never less reliable than naming a slug. The one case that raises is MELLONTOKEN_MODEL=auto as well, where there is no own model to keep.

Upgrading to 0.5.0

ask(..., llm=True) is unchanged and still does what it says. What changed is underneath it: the client now sends classifier="llm", where it used to send local=False, because the gateway replaced a boolean with a named choice — a boolean never said what would read the prompt instead.

If you passed local= directly, it is gone. It was never documented here, but it was reachable: local=True becomes the default (send nothing), and local=False becomes llm=True.

Upgrade the client if you use llm=True. Against a current gateway, 0.4.0's llm=True puts nothing on the wire, and the gateway now reads that as its own default — the local classifier. It will not fail; it will quietly give you the cheaper read. 0.5.0 says what it means.

web_search=True no longer narrows which model answers. The gateway runs the search itself rather than asking the provider, so the request needs a model that can call a function instead of one with a search index of its own — which is nearly all of them. A research question is now routed on the work it is, not on who happens to be able to look things up.

On model="auto" the flag is an override rather than the trigger: the router reads the need off the prompt, so a question that plainly wants current information searches without being told to. Pass web_search=False to forbid it.

Answer.web_search_calls says how many searches ran. Read it next to input_tokens: the fee is a tenth of a cent, and what searching costs is the results arriving as input.

Configuration

From the environment:

or in code:

mellontoken.configure(base_url="http://gateway:8000/api/v1", api_key=key, model="gpt-5.6-luna")

MELLONTOKEN_API_KEY is the only variable most callers set. The default base URL is the dev gateway, which resolves on the internal network only — from outside it, set MELLONTOKEN_BASE_URL yourself. Running the stack locally, that is http://localhost:8000/api/v1, or http://host.docker.internal:8000/api/v1 from inside a container, where localhost is the container rather than the host.

Requests go out as streams, and are reassembled here. ask still returns a finished answer — nothing about the API changes — but a blocking completion sends no byte until the model is done, and every proxy in front of the gateway drops a connection that has been silent for 60s. CloudFront does, and answers its own HTML error page; so does nginx, by default. A four-minute extraction never survived either. Streaming does, because the gateway starts the response within 20s whatever the model is doing and ticks a keep-alive every ten seconds after that — so MELLONTOKEN_TIMEOUT is the wall-clock ceiling again, rather than whatever the nearest intermediary allows.

What ask does that a bare HTTP call does not

• Sends the schema twice, as response_format and as prompt text. The request field is only honoured where the provider can enforce it; Gemini accepts a schema it will then ignore, and the field names in the prompt are what stop it inventing col_idx for col.
• Reads JSON out of a fenced reply, and wraps a bare list under the schema's single array field when that is unambiguous.
• Retries the shape, not the wire. A 401/402/403/404 is final — a key, a budget, a permission, a missing model — and raises immediately. A reply that did not parse is retried three times.
• Logs the cost of every call, which is the reason to route through the gateway at all: mellon.cost_usd is the only per-call price any provider path reports.
• Sends model="auto" straight through, and lets the gateway route it. This client used to resolve it first against /api/auto/preview and then send the slug that named — two round trips for one logical request. That endpoint still exists, for callers that want a decision without a completion; this path no longer uses it. Routing happens once per ask: the first reply names the model it chose and the shape-retries are pinned to it, so a reply that did not parse is a reason to ask the same model again rather than to pay three classifiers a second time.

Install

Built from this directory, which lives in the gateway's own repo so the client and the endpoint version together:

uv build            # -> dist/mellontoken-0.1.0-py3-none-any.whl