dispatch-relay

A provider-agnostic LLM layer with three injected seams. Resolve a model, dispatch a call across any provider, and relay usage to a sink your application owns — instead of the library tracking it for you. Pure-stdlib core, zero runtime dependencies.

pip install dispatch-relay

Who it's for: anyone running more than one LLM provider who wants one consistent dispatch + usage-attribution surface, with the host application in control of config resolution, usage recording, and the actual transport. The "relay, not track" name is the contract: usage is relayed to your sink (a database, a log, nothing) — the library never decides where it lands.

This is the dependency-light foundation increment: the three injected-interface seams + the pure cost model. (Caching and the higher-level façade arrive in a later increment and bring langchain-core etc. with them; this increment is pure-stdlib.)

Renamed from omega-llm. import omega_llm still works as a deprecated alias that re-exports dispatch_relay (with a DeprecationWarning) — migrate to import dispatch_relay.

The 3 injected seams (dispatch_relay.interfaces)

Each is a @runtime_checkable typing.Protocol (structural typing — a host satisfies the contract WITHOUT importing this library) + a dependency-light default impl.

cache_read and cache_creation are separate fields on UsageSink.record and on UsageRecord — summing them undercounts Anthropic. billing marks the lane: "metered" ($-tracked SDK) vs "subscription" ($0).

Value types & core-owned facts (dispatch_relay.core)

@dataclass(frozen=True)
class UsageRecord:  # input_tokens, output_tokens, cache_read=0, cache_creation=0, model=""
@dataclass(frozen=True)
class LLMResponse:  # text, usage: UsageRecord | None, raw: Any

The provider-facts live in dispatch_relay.core (one place, never duplicated per backend):

• DEFAULTS: dict[str, str] — the abstract-key → model-id table. The core passes default=DEFAULTS[key] into ConfigSource.resolve.
• extract_usage(provider, raw) → UsageRecord | None — the single place that knows each provider's usage-from-raw shape. Anthropic dual-path: prefer raw.response_metadata["usage"] (the uncached remainder), fall back to raw.usage_metadata only if absent (using the wrong one double-counts). The model name comes from raw.response_metadata["model_name"] (both Anthropic and Gemini surface it there — a real LangChain AIMessage has no top-level .model attribute), falling back to "". Returns None when no usage metadata is present.
• resolve_usage(response, provider, model) → UsageRecord | None — the locked reconciliation rule: resolve response.usage if response.usage is not None else extract_usage(provider, response.raw), then stamp the authoritative model — the dispatch call knows the configured model, so the dispatch-arg model always wins over whatever the raw echoed (via dataclasses.replace). Returns None unchanged when there's no usage (the subscription lane). LLMResponse.usage is a real escape hatch — a backend MAY pre-populate it; else the core extracts.

Both shipped backends return LLMResponse(usage=None); the core extracts usage. The DefaultDispatchBackend derives text from raw.content: a str passes through; an Anthropic content list has its type=="text" blocks joined (non-text blocks skipped); anything else falls back to str(raw). That fallback is only the default backend's degenerate case — real subscription backends (raws are dicts, not strings) construct text explicitly and pass usage=None with billing="subscription".

The pure cost model (dispatch_relay.cost)

estimate_cost(*, prompt, tier="flash", provider="gemini", output_tokens_max=1024, cache_hit_ratio=0.0, role="agents") -> dict — a single source of cost truth. Pricing tables for Gemini / Anthropic / OpenAI, the Gemini Flex 50% rebate gate, Anthropic + OpenAI cache-ratio math. Zero deps.

Usage

from dispatch_relay import estimate_cost, DefaultConfigSource, DEFAULTS

DefaultConfigSource().resolve("gemini_flash", "council", DEFAULTS["gemini_flash"])
# -> "gemini-2.5-flash"  (env GEMINI_FLASH_MODEL wins if set)
estimate_cost(prompt=10_000, tier="sonnet", provider="anthropic", output_tokens_max=512)

Authors

Pierre Samson and Claude. MIT licensed.