mellontoken

Client for the mellontoken gateway.

import mellontoken

answer = await mellontoken.ask("Summarise this tender in one line.")
parsed = await mellontoken.ask(document_text, schema=DocExtraction)
full   = await mellontoken.ask_full(document_text, schema=DocExtraction)
full.cost_usd     # "0.00059550"
full.model        # the model that actually served it

Letting the gateway pick the model

answer = await mellontoken.ask(prompt, model="auto")
answer = await mellontoken.ask(prompt, model="auto", tier="high")   # bound the spend

full = await mellontoken.ask_full(prompt, model="auto")
full.route.model_slug    # what it picked
full.route.task          # the grid row it read
full.route.explanation   # why, in a paragraph you can argue with

One command, and one request. model="auto" goes to the gateway as the model name; the gateway classifies the prompt on three axes (task, effort, capability), reads the resulting grid cell, and serves the answer from whatever that cell names — all inside the same call. ask_full reports the decision on Answer.route, so routing is legible after the fact without a second call to ask about it.

All three axes come from one classifier call, billed on top of the completion. It reads roughly 13–20k input tokens however short your prompt, so it is not free: full.route.cost_usd is what it was billed, and full.total_cost_usd is the call's whole price. A prompt the gateway has read before reuses the stored reading and costs nothing (route.from_cache). llm= is still accepted and ignored.

It lands on the ledger as a kind=classifier row, so the spend is visible rather than hidden. A service sending the same shape of prompt a thousand times should read full.route.model_slug once and then name that slug: the decision is a property of the workload, not of each call.

tier= bounds what the router may spend and outranks the effort classifier, because a tier is a statement about your budget rather than a property of the text. Set it per call, or once via MELLONTOKEN_TIER. It is only read on the auto path — with a named model it is logged as ignored rather than silently dropped.

A router that resolves nothing is not an error. When the capability gate rejects every candidate in the cell the gateway answers 422 with the reason; ask logs it and retries against MELLONTOKEN_MODEL, which is what that error advises — keep your own model, so auto is never less reliable than naming a slug. The one case that raises is MELLONTOKEN_MODEL=auto as well, where there is no own model to keep.

The cost, when it comes later

Through the gateway's Envoy path a completion is billed a moment after its stream ends, so the answer arrives with cost_usd == "?". Ask for the bill when you want the figure:

full = await mellontoken.settled(full)   # waits up to 10s; wait= to change it
full.cost_usd                            # "0.00059550", or still "?" if it was never billed

An answer whose cost is already known comes back unchanged, without a request.

Upgrading to 0.7.0

• Answer.cost_usd is now everything the call paid for in completions. A reply that failed to parse and was retried, and every turn of a tools= loop, used to be billed but missing from the figure; they are summed now. Answer.attempts says how many completions that was.
• Routing has a price. Route.cost_usd is the classifier call, and Answer.total_cost_usd adds it to cost_usd. Needs a gateway that reports it; an older one leaves both as "?" rather than guessing.
• Answer.route survives a retry. It used to come back None whenever the answer was a second attempt, losing the routing decision.
• Field descriptions reach the model. The schema written into the prompt keeps Field(description=...) — the only place a provider that ignores response_format sees it. Titles are still dropped.
• The environment is read at call time. load_dotenv() after import mellontoken now works; configure() still wins over it.

Upgrading to 0.5.0

ask(..., llm=True) is unchanged and still does what it says. What changed is underneath it: the client now sends classifier="llm", where it used to send local=False, because the gateway replaced a boolean with a named choice — a boolean never said what would read the prompt instead.

If you passed local= directly, it is gone. It was never documented here, but it was reachable: local=True becomes the default (send nothing), and local=False becomes llm=True.

Upgrade the client if you use llm=True. Against a current gateway, 0.4.0's llm=True puts nothing on the wire, and the gateway now reads that as its own default — the local classifier. It will not fail; it will quietly give you the cheaper read. 0.5.0 says what it means.

web_search=True no longer narrows which model answers. The gateway runs the search itself rather than asking the provider, so the request needs a model that can call a function instead of one with a search index of its own — which is nearly all of them. A research question is now routed on the work it is, not on who happens to be able to look things up.

On model="auto" the flag is an override rather than the trigger: the router reads the need off the prompt, so a question that plainly wants current information searches without being told to. Pass web_search=False to forbid it.

Answer.web_search_calls says how many searches ran. Read it next to input_tokens: the fee is a tenth of a cent, and what searching costs is the results arriving as input.

Configuration

From the environment:

or in code:

mellontoken.configure(base_url="http://gateway:8000/api/v1", api_key=key, model="gpt-5.6-luna")

Read when a call is made rather than at import, so loading a .env after import mellontoken works; anything set with configure() wins.

MELLONTOKEN_API_KEY is the only variable most callers set. The default base URL is the dev gateway, which resolves on the internal network only — from outside it, set MELLONTOKEN_BASE_URL yourself. Running the stack locally, that is http://localhost:8000/api/v1, or http://host.docker.internal:8000/api/v1 from inside a container, where localhost is the container rather than the host.

Requests go out as streams, and are reassembled here. ask still returns a finished answer — nothing about the API changes — but a blocking completion sends no byte until the model is done, and every proxy in front of the gateway drops a connection that has been silent for 60s. CloudFront does, and answers its own HTML error page; so does nginx, by default. A four-minute extraction never survived either. Streaming does, because the gateway starts the response within 20s whatever the model is doing and ticks a keep-alive every ten seconds after that — so MELLONTOKEN_TIMEOUT is the wall-clock ceiling again, rather than whatever the nearest intermediary allows.

What ask does that a bare HTTP call does not

• Sends the schema twice, as response_format and as prompt text. The request field is only honoured where the provider can enforce it; Gemini accepts a schema it will then ignore, and the field names in the prompt are what stop it inventing col_idx for col.
• Reads JSON out of a fenced reply, and wraps a bare list under the schema's single array field when that is unambiguous.
• Retries the shape, not the wire. A 401/402/403/404 is final — a key, a budget, a permission, a missing model — and raises immediately. A reply that did not parse is retried three times.
• Logs the cost of every call, which is the reason to route through the gateway at all: mellon.cost_usd is the only per-call price any provider path reports.
• Sends model="auto" straight through, and lets the gateway route it. This client used to resolve it first against /api/auto/preview and then send the slug that named — two round trips for one logical request. That endpoint still exists, for callers that want a decision without a completion; this path no longer uses it. Routing happens once per ask: the first reply names the model it chose and the shape-retries are pinned to it, so a reply that did not parse is a reason to ask the same model again rather than to pay three classifiers a second time.

Install

Built from this directory, which lives in the gateway's own repo so the client and the endpoint version together:

uv build            # -> dist/mellontoken-0.7.0-py3-none-any.whl