llmkit

A thin, opinionated, local-first layer over LiteLLM (with instructor for structured output). It gives an application one provider-agnostic call surface across OpenRouter, Google AI Studio, Google Vertex AI, Anthropic, OpenAI, DeepSeek, AWS Bedrock, and local Ollama, with validated structured output, per-provider rate limiting (concurrency on by default; optional requests-/tokens-per-minute), agent-readable per-call logging, and transient-error retries on by default — all out of the box.

LiteLLM is the implementation of the HTTP providers; llmkit owns the ergonomic call surface, the structured-output mode pinning, the rate-limit policy, and the logging convention. It is not a gateway and does not reimplement transport — that is solved, and reimplementing it is the thing this library deliberately does not do.

Why llmkit

• Structured output that actually validates. Each provider is pinned to an explicit, live-measured structured-output mode (never instructor's auto-Mode.TOOLS, which silently regresses Gemini to empty shapes), and instructor's in-call validation-retry repairs malformed JSON. You pass a Pydantic model; you get a validated instance back. (A completion truncated by the output-token limit is the one thing never re-asked — it fails fast as OutputLimitError; see Retries.)
• Provider switching is config, not code. OpenRouter / Google AI Studio / Google Vertex AI / Anthropic / OpenAI / DeepSeek / AWS Bedrock / Ollama behind one Provider enum and one LLMClientConfig. Call sites never change when you switch. The same Gemini models are reachable two ways — AI Studio (a bearer key) or Vertex AI (Google Cloud, with a vertex_location data-residency control) — just like Claude is reachable direct or via Bedrock.
• Logging tuned for coding agents. Every call is logged verdict-first (see below) — the design assumption is that the reader is usually an LLM coding agent debugging a run, not a dashboard.
• Local-first, zero infra. The default sink writes plain files to a directory. No collector, no account, no network. A pluggable LogSink lets you ship records anywhere later without touching call sites.

These four are the headline; PRINCIPLES.md states the full set of design principles behind the library.

Install

uv add omg-llmkit          # or: pip install omg-llmkit

The distribution is published as omg-llmkit (the bare llmkit name was already taken on PyPI), but the import name is just llmkit:

import llmkit

You install omg-llmkit but import llmkit — that split trips a natural post-install smoke test. A mistaken import omg_llmkit (the install name) raises a clear one-line redirect to import llmkit, not a bare ModuleNotFoundError that leaves you guessing.

llmkit.__version__ reports the installed version. It is read from the omg-llmkit distribution's metadata rather than hardcoded, so it cannot drift from what you installed; a source tree with no installed metadata reports "0.0.0+unknown" instead of raising at import.

Requires Python ≥ 3.12.

The core install routes OpenRouter, Google AI Studio, Anthropic, OpenAI, DeepSeek, and Ollama with no extra dependencies. Two providers gate their dependencies behind opt-in extras so hosts pay only for what they call:

pip install "omg-llmkit[bedrock]"    # Claude-on-Bedrock (boto3 for SigV4 signing)
pip install "omg-llmkit[vertex]"     # Gemini via Google Cloud Vertex AI
pip install "omg-llmkit[all]"        # both of the above

Bedrock's [bedrock] extra pulls only boto3 (LiteLLM signs Bedrock requests with AWS SigV4 through it), and constructing the BedrockProvider without it raises a clear install omg-llmkit[bedrock] error at construction, not a cryptic failure on the first call. Vertex is the same shape: its [vertex] extra pulls only google-auth (LiteLLM mints the Vertex OAuth token through it), and constructing the VertexProvider without it raises a clear install omg-llmkit[vertex] error — a non-Vertex host takes on no Google dependency. Direct Anthropic routing needs no SDK at all: LiteLLM speaks the Anthropic HTTP API itself, so the [anthropic] extra that existed through 0.6.x is gone.

Quick start

from pydantic import BaseModel
from llmkit import (
    LLMClientConfig,
    Provider,
    configure_llm_client,
    structured_llm_call,
)

# Point the library at a provider once, at startup.
configure_llm_client(lambda: LLMClientConfig(
    provider=Provider.OPENROUTER,
    model="google/gemini-2.5-flash",
    api_key="sk-or-...",
))

class Summary(BaseModel):
    title: str
    bullets: list[str]

result: Summary = await structured_llm_call(
    prompt="Summarize the attached report.",
    output_schema=Summary,
    feature="reports",      # groups calls in the logs
    label="exec_summary",   # names this specific call in the logs
)

The public call surface:

prompt is typed str | list[Message] on all five. A plain string is sent as-is; the list form is a list of llmkit.Message — a TypedDict whose role is "system", "user", or "assistant" and whose content is either a string or a list of content-part dicts, the multimodal shape LiteLLM accepts ({"type": "text", ...}, {"type": "image_url", ...}), forwarded verbatim. Message is exported so your own prompt builders can be annotated against it rather than against the transport's wire shape: an unknown key ({"roel": ...}) or a mistyped role is a type error, and multimodal content type-checks instead of being rejected.

Type-check migration. prompt was previously str | list[dict[str, str]], so a call site passing a variable annotated list[dict[str, str]] now fails the type check — re-annotate it list[Message]. Inline message-dict literals are unaffected, and runtime behaviour is identical either way (a TypedDict is a plain dict).

Deprecated alias. stream_text_with_log is the old name for text_llm_call_stream; it still works (same signature and behaviour) but warns DeprecationWarning and is removed in 1.0. Switch the call.

Two defaults worth knowing up front.

• temperature defaults to 0.2 — biased toward deterministic output. A creative caller must override it explicitly (e.g. temperature=1.0); it is otherwise quietly conservative.
• Any call takes a per-call provider= override — route a single call through a different provider family, model, or credential without touching the global configure_llm_client(...) registration. See Constructing a provider for a per-call override.

Reusing call options

The call functions (structured_llm_call, structured_llm_call_sync, text_llm_call, text_llm_call_sync, and text_llm_call_stream) take a block of per-call keyword arguments. When a feature module makes many calls with the same settings, repeating that block at every site is noise. Build an LLMCallOptions once and pass it as options=:

from llmkit import LLMCallOptions, structured_llm_call

# Built once per feature module.
RISK_OPTS = LLMCallOptions(
    temperature=0.0,
    model="gemini-2.5-flash",
    reasoning_effort="high",
    max_tokens=2048,
)

async def extract(prompt: str) -> RiskRegister:
    return await structured_llm_call(
        prompt, RiskRegister, feature="extraction", options=RISK_OPTS
    )

LLMCallOptions is frozen and carries any subset of temperature / model / max_tokens / reasoning_effort / retry / provider. Every field is optional and unset by default — an unset field defers to the call's keyword (and through it to the configured client), so a partially-filled LLMCallOptions only supplies the fields you set.

feature is intentionally not part of LLMCallOptions. It stays a required per-call keyword as a telemetry forcing function: it scopes the per-call log filename and the index.jsonl grouping operators grep, so it must be a conscious choice at each call site rather than something defaulted-away into a shared object.

The flat-keyword path is unchanged — pass no options and nothing about existing calls changes.

Call-vs-config precedence

model and reasoning_effort are dual-homed: they can be set both on LLMClientConfig (the app-wide default) and on the call surface. The precedence, lowest to highest, is:

config < options < explicit per-call keyword

So a value passed directly as a keyword wins — any passed value, including None or one equal to the documented default. The call functions' mergeable keywords default to the UNSET sentinel rather than to real values, so "was it passed" is a structural fact, never inferred by comparing values: structured_llm_call(..., temperature=0.2, options=LLMCallOptions(temperature=0.9)) runs at 0.2, and an explicit model=None forces the provider/config default even when options carries a model. An LLMCallOptions field sits between the keyword and the config; when neither the keyword nor options supplies a value, the true default applies (DEFAULT_TEMPERATURE, DEFAULT_RETRY_POLICY, or the configured LLMClientConfig resolution for model/reasoning_effort). An unset LLMCallOptions field never overrides config — only a field you explicitly set on the options participates.

Unset (the type) and UNSET (the value) are exported for one idiom: your own typed wrapper can declare temperature: float | Unset = UNSET and forward it unconditionally, letting llmkit resolve "not passed" instead of re-inventing the sentinel. Compare with is/is not only — never truthiness.

Contracts as JSON-schema dicts

If your structured-output contract is a JSON-schema dict — typically because the same schema is shared with a Node backend or a frontend — model_from_json_schema(schema) converts it to a Pydantic model at runtime, so you don't hand-write the converter (and re-discover its footguns). Build the model once and reuse it; structured_llm_call stays Pydantic-model-only and takes the result as output_schema.

from llmkit import model_from_json_schema, structured_llm_call

INVOICE_SCHEMA = {                       # shared with Node / the frontend
    "title": "Invoice",
    "type": "object",
    "properties": {
        "id": {"type": "string"},
        "total": {"type": "number"},
        "status": {"enum": ["open", "closed", "void"]},
        "note": {"type": ["string", "null"]},          # optional, nullable
        "lines": {"type": "array", "items": {"$ref": "#/$defs/Line"}},
    },
    "required": ["id", "total", "status", "lines"],
    "$defs": {
        "Line": {
            "type": "object",
            "properties": {"sku": {"type": "string"}, "qty": {"type": "integer"}},
            "required": ["sku"],
        }
    },
}

Invoice = model_from_json_schema(INVOICE_SCHEMA)   # build once, at import

result = await structured_llm_call(
    prompt="Extract the invoice.",
    output_schema=Invoice,                         # reuse on every call
    feature="billing",
)

Supported subset (anything outside it raises a clear ValueError naming the construct): object with properties and a required array; scalars (string / integer / number / boolean, plus null / nullable); array with items (including arrays of objects); enum (string or integer members); nested objects inline or via local $ref (#/$defs/...); and additionalProperties as true / false / absent (a typed additionalProperties map is rejected). A non-required field becomes an optional defaulting to None, and the generated model's model_dump / model_dump_json drop a None left in an optional field by default — so an omitted optional is absent, not "field": null (which would fail downstream re-validation against the same schema). The drop is scoped to optionals: a required-but-nullable field explicitly set to None is kept. Pass exclude_none=False to keep every null, or exclude_none=True to drop them all. A title-less schema still gets a valid default class name (JsonSchemaModel); pass name= to set it explicitly. Generated models default to extra="forbid", so a response carrying a key not in the schema is rejected rather than silently kept — for an LLM output contract you want a hallucinated extra field to fail loudly (stricter than JSON Schema's permissive additionalProperties default); "additionalProperties": true opts an object into extra="allow" (extra keys accepted and kept), while false or absent stays strict. An "type": "object" with no properties — properties absent or an explicit empty {} — raises rather than silently building a zero-field model that rejects every real response; set "additionalProperties": true for an intentionally free-form object.

Want plain data back, not a model instance? Call .model_dump() on the result — it inherits the optional-None drop above, so the dict matches the schema:

Person = model_from_json_schema(person_schema)   # build once, at import

result = await structured_llm_call(prompt, Person, feature="extraction")
data = result.model_dump()                        # {"name": "Ada", "age": 36}

Schema constraints

model_from_json_schema carries a small, fixed set of per-field JSON-schema constraints through to the generated Pydantic Field, so the model validates value bounds, not just shape. The supported set is exactly:

Score = model_from_json_schema(
    {
        "type": "object",
        "properties": {"score": {"type": "integer", "minimum": 1, "maximum": 5}},
        "required": ["score"],
    }
)
Score(score=3)   # ok
Score(score=6)   # raises pydantic.ValidationError

Bounds are resolved through $ref chains of any depth and through nullable wrappers, so a constraint declared inside a $def (even several $ref hops deep) or on the non-null branch of a nullable field is still enforced (and null itself still passes for a nullable field). The same resolution carries a $def's description through, including behind a nullable wrapper — a nullable $ref field surfaces the target's model-facing guidance just like a bare $ref does.

Keywords that sit beside a $ref are handled by kind. Metadata and the bounds above merge with the referenced schema, the outer (property-level) value winning on conflict — so {"$ref": "#/$defs/Count", "minimum": 5} keeps the bound. A keyword that would redefine the reference's structure (type, enum, items, properties, required, additionalProperties, anyOf, oneOf) is a JSON-Schema conjunction a merge cannot express, so it is rejected unless it restates the target's own value — an enum beside a $ref raises a clear ValueError rather than silently widening the field to an unconstrained scalar. Move such a keyword into the referenced $def, or inline the schema.

Subschema applicators are rejected everywhere, not only beside a $ref. allOf, not, if / then / else, dependentSchemas, dependentRequired, propertyNames, patternProperties, prefixItems, contains, unevaluatedProperties, and unevaluatedItems constrain an instance by composition, and a generated field is one annotation plus the bounds in the table above — so an applicator has nowhere to land. Each raises a ValueError naming the keyword and its path, at every site: a property, an array's items, a nested object, the body of a $def reached through a $ref, the non-null branch of a nullable anyOf, and the root object.

model_from_json_schema(
    {
        "type": "object",
        "properties": {"score": {"type": "integer", "allOf": [{"maximum": 10}]}},
        "required": ["score"],
    }
)
# ValueError: Unsupported keyword 'allOf' at '$.score': subschema applicators ...

Rejecting is not pedantry — dropping one is wrong in both directions. A dropped allOf bound makes the model accept a value the schema forbids; a dropped prefixItems changes what the sibling items means (Draft 2020-12 reads it as "every element after the prefix"), so the model rejects a response the schema permits and the spurious ValidationError is paid for in retries. Either express the constraint with a supported keyword, or validate it outside the generated model. The nullable spellings — anyOf / oneOf with a single non-null branch plus a null branch — are unaffected and still build.

A propertyless object is rejected the same way: {"type": "object"} with no properties — or an explicit empty {"properties": {}} — raises at translation time (it would otherwise build a zero-field model that rejects every real response), unless it opts into open-ended keys with "additionalProperties": true.

One form caveat: exclusiveMinimum / exclusiveMaximum are recognised in their numeric (Draft 2020-12) form only. The Draft-4 / OpenAPI-3.0 boolean form ("exclusiveMinimum": true qualifying a sibling "minimum") is not recognised and is dropped — the bound is enforced as the sibling's inclusive minimum/maximum. If your schema comes from an OpenAPI 3.0 document, rewrite exclusive bounds in the numeric form.

Leaf constraints outside the table above are silently dropped — pattern, format, multipleOf, uniqueItems, const, and the rest are not enforced. This is deliberate: partial enforcement that looks complete is worse than none. If a schema relies on one of those, validate it elsewhere. The drop is scoped to per-value keywords like these; a structural construct outside the supported subset — a subschema applicator, a multi-variant union, a typed additionalProperties map — raises instead of vanishing, because losing one silently changes the shape the model validates rather than merely leaving a value unchecked.

Rate limiting

Rate limiting is on by default, scoped per provider (keyed by the effective provider name, matching how logging records it), across three independent dimensions:

• Concurrency — on by default, default cap 8 concurrent calls per provider: enough headroom for the fan-out workloads consumers actually run, while still bounding a self-inflicted burst; lower it for a tightly-metered account, raise it for a local Ollama server. This cap is also adaptive by default — it backs off when a provider signals overload and recovers automatically (see Adaptive concurrency below). The cap binds async callers and the *_sync wrappers alike — and because every *_sync call runs on llmkit's single persistent event loop, a thread-pool fan-out of sync calls shares one per-provider cap. One caveat follows from the concurrency gate being bound to a single loop: a process that drives calls on more than one loop (e.g. running the async call functions on its own event loop and the *_sync wrappers on the persistent loop) caps each loop's population independently, so it can momentarily hold up to loops × the cap per provider; the adaptive limit and the RPM/TPM budgets are shared across all of them.
• Requests per minute (RPM) — opt-in, off by default. A per-provider request-rate ceiling.
• Tokens per minute (TPM) — opt-in, off by default. A per-provider token-rate ceiling, debited by each call's measured token usage.

configure_rate_limit(max_concurrent=..., enabled=..., rpm=..., tpm=..., adaptive=..., breaker=...) sets them; get_rate_limit_config() reads back the effective enabled / max_concurrent / rpm / tpm / adaptive / breaker (handy to log or assert at startup); configure_llm_logging(sink) swaps the log sink (below). The breaker switch arms the opt-in per-provider circuit breaker.

from llmkit import configure_rate_limit

# Stay under a metered account's published per-minute limits:
configure_rate_limit(rpm=3_500, tpm=2_000_000)

RPM and TPM are opt-in because — unlike concurrency, which has a universally sane default of 8 — the right per-minute number is the metered limit of your account, with no safe default to assume. Leaving them unset sends a request byte-identical to the pre-feature behaviour (no throttle on those dimensions). The binding limit on a metered cloud account is usually RPM/TPM rather than concurrency, so a migrator coming from a requests-per-minute knob should set rpm= here — the concurrency cap does not stand in for an RPM limit (the two limit different things, and an old RPM tuning otherwise goes inert). Both use a per-provider token bucket, which tolerates a small burst above the configured ceiling and then smooths to the sustained rate. That burst is deliberately small — min(max_concurrent, rpm) requests for RPM, roughly one second of tokens for TPM — not a full minute's quota. Against a provider that enforces a strict fixed minute window, the burst is the worst-case overshoot, so its relative size scales with your limits: with the default max_concurrent=8 it is negligible at rpm=3_500 (~0.2%) but a meaningful fraction of a small limit (8 extra requests on rpm=50 is 16%). A tightly-metered account should lower max_concurrent (which shrinks the RPM burst with it) or set rpm= a little below the published number to leave headroom. When the RPM ceiling does make calls wait, they are admitted in arrival order (FIFO): a late arrival cannot jump the queue ahead of a caller already waiting on the same event loop, so no caller on that loop is starved under sustained saturation. The queue is per loop, exactly like the per-loop concurrency caveat above — a process that drives the same provider across more than one loop (the async call functions on its own loop and the *_sync wrappers on the persistent loop) gets per-loop FIFO rather than one global arrival order across them. (Host code that joins the limiter directly through the synchronous rate_limit_acquire_sync orders on its own independent ticket queue as well.) The aggregate rate stays exact regardless of how the queues interleave. (A streamed call usually reports no token usage, so it does not debit TPM — consistent with cost being None for streamed calls.)

Adaptive concurrency

The per-provider concurrency limit is adaptive by default: when a provider pushes back with an overload signal (HTTP 429 / 503 / 529), llmkit lowers that provider's in-flight limit, and raises it back toward max_concurrent once the provider stops pushing back — a TCP-style additive-increase / multiplicative-decrease (AIMD) loop, with zero per-account tuning. It discovers a safe concurrency under a sustained-overload window that a fixed cap cannot ride out, and is the library-side generalization of a hand-tuned RPM ceiling.

It only ever lowers the limit below max_concurrent, never above, so a provider that never throttles behaves identically to a fixed cap — adaptive concurrency is a no-op until the provider actually pushes back. The decrease only fires when you were genuinely saturated (running at the limit when the throttle arrived), so an isolated throttle received while running well under the cap doesn't penalise a healthy provider; recovery is paced by wall-clock time, so it is bounded regardless of how slowly your workload trickles. It is, honestly, a trade, not a free win: during a real overload it deliberately runs narrower (which is correct), and the residual cost is a brief reduced-width window after the provider recovers. Turn it off with configure_rate_limit(adaptive=False) to pin the limit at max_concurrent. (Adaptive concurrency is also a strong complement to Retry-After, below — the retry waits as the provider directs, while the fleet as a whole backs off.)

from llmkit import configure_rate_limit

configure_rate_limit(max_concurrent=8)             # adaptive (the default)
configure_rate_limit(max_concurrent=8, adaptive=False)  # fixed cap, pre-feature behaviour

Observing backpressure

Install a backpressure_callback to see the adaptive limiter move in real time — for metrics, a budget-visibility dashboard, or just logging. The callback receives a BackpressureEvent(provider, old_limit, new_limit, reason) each time a provider's limit changes (reason is "throttle" or "recover") or — when the circuit breaker is armed, below — the breaker changes state ("breaker_open", "breaker_half_open", "breaker_closed"). It is read from a context variable — like retry_progress_callback — so it propagates across the run_sync boundary; install it once around your fan-out.

from llmkit import backpressure_callback, BackpressureEvent

def on_backpressure(event: BackpressureEvent) -> None:
    print(f"{event.provider}: {event.old_limit} -> {event.new_limit} ({event.reason})")

with backpressure_callback(on_backpressure):
    ...  # calls here report every adaptive limit change to on_backpressure

A callback that raises is swallowed and logged — observability can never break a call.

Circuit breaker

Adaptive concurrency drives a struggling provider's limit toward 1; the circuit breaker is the "limit is effectively 0 while the provider is down" case it cannot express. It is opt-in and off by default — arm it with configure_rate_limit(breaker=True). Once a provider's throttle rate over a rolling window (the last 20 real outcomes, at least half of them throttled) trips it, llmkit fails fast for that provider: every call raises CircuitOpenError immediately — holding no concurrency slot and deducting no RPM token — for a cooldown, instead of letting each call burn its retry budget into the storm (the load-adding pathology of retries under sustained overload). After the cooldown a single probe tests recovery: a clean success closes the breaker, any failure re-opens it for another cooldown.

from llmkit import configure_rate_limit, CircuitOpenError, structured_llm_call

configure_rate_limit(breaker=True)  # opt in; off by default

try:
    result = await structured_llm_call(prompt, MySchema, feature="extract")
except CircuitOpenError as exc:
    ...  # the breaker is open for exc.provider — fall back fast, don't hammer it

It is off by default on purpose. Adaptive concurrency is a safe default because it only ever reduces load below your cap; the breaker changes the contract — it flips "eventually succeeds" into "fails fast" — so the host decides. CircuitOpenError is in LLM_RECOVERABLE_ERRORS (a host that already writes except LLM_RECOVERABLE_ERRORS to degrade on a 503 keeps catching it, and falls back fast) but the library never retries it — retrying a circuit you already know is open would defeat the point. Each provider has its own breaker, and the internal thresholds (window 20, trip fraction 0.5, cooldown 30s) are library-owned mechanism, not knobs — the public surface stays "three numbers and three switches" (enabled / adaptive / breaker).

Joining the global rate limit directly

llmkit's own call functions already pass every provider call through the global, per-provider limit (concurrency on by default; RPM/TPM when configured). If your app issues provider calls outside those functions — for example a LangChain chat-model wrapper that calls the provider itself — you can join the same per-provider budget by hand with the module-level acquire functions:

from llmkit.rate_limiting import (
    rate_limit_acquire_async,
    rate_limit_acquire_sync,
)

# Async path (e.g. an async _agenerate):
async with rate_limit_acquire_async("openai") as slot:
    response = ...  # one slot held against openai's budget
    slot.record_tokens(response.usage.total_tokens)  # debits TPM (no-op when off)

# Sync path (e.g. a synchronous _generate / _stream):
with rate_limit_acquire_sync("openai") as slot:
    response = ...  # one slot held against openai's budget
    slot.record_tokens(response.usage.total_tokens)

The argument is the provider name (provider.name, e.g. "openai", "ollama"); each provider has an independent budget on every dimension. Each context manager yields a RateLimitSlot; call its record_tokens(...) once you know the call's token usage to debit the TPM budget (a no-op when TPM is off). Both are no-ops when rate limiting is disabled, and they share the exact throttle llmkit's own call paths use, so a hand-joined slot counts against the same budgets.

To check whether limiting is currently active, read the effective config rather than reaching into the limiter:

from llmkit.rate_limiting import get_rate_limit_config

if get_rate_limit_config().enabled:
    ...

get_rate_limit_config().enabled is the public replacement for the old GlobalRateLimiter.is_enabled() check; GlobalRateLimiter itself is no longer part of the headline surface (it remains importable from llmkit.rate_limiting for internal use).

Logging: agent-readable by default

LocalYamlLogSink (the default) writes two things to the log directory:

1. One YAML file per call, laid out verdict-first. The file opens with a one-line # header — ok/ERROR, feature/label, resolved model, schema, duration, approximate cost — so head -1 *.yaml triages a whole run (the second header line carries the timestamp plus call=<id> attempt=<n>, so retries of one logical call are joinable from the file heads alone). Small metadata is next; the large response and prompt blobs are last, so the head of the file is the whole story for most reads.
2. A compact append-only index.jsonl — one JSON line per call (file, timestamp, feature, label, model, provider, schema, call_id, attempt, duration, queue wait, cost, error). Cross-call questions — "which calls errored / were slowest / most expensive / the last call for feature X" — are a single small scan instead of globbing and parsing every YAML.

# ok | reports/exec_summary | google/gemini-2.5-flash | Summary | 1840ms | $0.0007
# 2026-06-05T14:22:31.004512 | call=9f3c21ab attempt=1

timestamp: '2026-06-05T14:22:31.004512'
feature: reports
label: exec_summary
model: google/gemini-2.5-flash
provider: openrouter
schema: Summary
call_id: 9f3c21ab54d64f1f8f2c14febc03a7d1
attempt: 1
temperature: 0.0
max_tokens: null
reasoning_effort: null
duration_ms: 1840.2
queue_wait_ms: 0.4
approximate_cost: 0.0007
error: null
response: ...
prompt: ...

approximate_cost is LiteLLM's per-response estimate for budget visibility — not a billing figure (and None when the provider does not report it, e.g. streamed calls). call_id is one id per logical call and attempt the 1-based attempt within it, so the N records a retried call produces join on call_id. duration_ms measures the whole attempt including queue_wait_ms — the time spent queued behind llmkit's own rate limiter — so provider latency is approximately duration_ms - queue_wait_ms (hook time and in-call schema-repair re-asks are also inside duration_ms). queue_wait_ms is float | None, not always a float: it is 0.0 when the limiter is disabled and None when the attempt failed before acquiring a slot, so a custom sink or index.jsonl parser has to handle the null rather than subtract it blindly.

Where the logs go

The default directory is resolved lazily at the first write and then frozen for the sink's lifetime (a mid-run chdir can't split logs):

1. LLMKIT_LOG_DIR, when set;
2. data/llm-logs/ under the nearest ancestor directory carrying a pyproject.toml or .git (nearest wins) — and when the sink creates that directory it also seeds a .gitignore, so prompt logs never land in your repository's history;
3. otherwise a per-user state directory ($XDG_STATE_HOME/llmkit/llm-logs on Linux, ~/Library/Logs/llmkit on macOS, %LOCALAPPDATA%\llmkit\logs on Windows) — never a CWD-relative path.

The first successful write logs one INFO naming the absolute directory and the retention policy. default_log_dir() returns the currently-resolved answer; an explicit LocalYamlLogSink(log_dir=...) is used as given, made absolute at construction so that a relative path (or a relative LLMKIT_LOG_DIR) names one directory for the sink's lifetime instead of following the process around. Pass configure_llm_logging(None) to disable logging entirely.

On POSIX, a sink-created directory is 0o700 and log files are 0o600 — prompt data is owner-only by default. A pre-existing directory is never re-chmodded: pre-create the directory yourself to share logs with other readers.

Retention: bounded by default

Long-running services no longer accumulate unbounded prompt data. By default the sink prunes per-call YAML files older than 30 days and rotates index.jsonl past 50 MiB to a date-stamped sibling (which ages out under the same policy). Housekeeping runs on the write path, throttled to once per hour, off the event loop.

LocalYamlLogSink(retention_days=None)                  # keep everything forever
LocalYamlLogSink(retention_days=7, max_index_bytes=None)  # tighter age bound, no rotation
LocalYamlLogSink(retention_days=0)                     # ValueError — see below

Each bound takes a positive integer or None; 0 and negatives raise ValueError at construction. 0 is rejected rather than quietly interpreted because it has two opposite plausible meanings — "keep nothing" (logrotate's rotate 0) and "no limit" (many SaaS retention settings) — and the destructive reading is what the mechanics would actually do: the prune cutoff would be now, so the first write would delete every log in the directory, including the file it had just written and whose path it returns. None is the only opt-out, for both bounds.

Sink I/O never runs on the event loop: writes are offloaded to a worker thread (the one deliberate exception is a stream abandoned mid-flight, whose record is written synchronously so it can't be lost while the generator unwinds).

Capturing call records

Every call function (structured_llm_call, structured_llm_call_sync, text_llm_call, text_llm_call_sync, and text_llm_call_stream) builds an LLMCallRecord and hands it to the configured log sink. A higher-level orchestrator that needs to cross-reference those calls — to total approximate cost, attribute spend per feature, or weave per-call traces — has two additive capture primitives, neither of which requires authoring a sink.

capture_llm_records() — records (cost / metadata). Wrap a scope to receive the LLMCallRecord for every call made inside it. Each record carries approximate_cost (a best-effort USD estimate, None when the provider doesn't report it), the resolved model/provider, duration_ms, error, and the rest — so a host gets cost and metadata without writing a custom sink. Capture is sink-independent: it works even with logging disabled (configure_llm_logging(None)), and crosses the run_sync sync bridge, so structured_llm_call_sync is captured exactly like the async path. One record is appended per attempt (retries each produce their own).

from llmkit import capture_llm_records, structured_llm_call

with capture_llm_records() as records:
    result = await structured_llm_call(prompt, MySchema, feature="extraction")

total_cost = sum(r.approximate_cost or 0.0 for r in records)

capture_llm_log_paths() — file paths. Wrap a scope to receive the per-call log-file path written by the configured file sink. Only a file sink (the default LocalYamlLogSink) yields a path; with a third-party sink, or with logging disabled, the list stays empty — reach for capture_llm_records() when you want cost/metadata regardless of the sink.

from llmkit import capture_llm_log_paths, structured_llm_call

with capture_llm_log_paths() as paths:
    result = await structured_llm_call(prompt, MySchema, feature="extraction")
# paths -> [PosixPath("data/llm-logs/...yaml"), ...]

Write your own LogSink

LogSink is a Protocol with a single, file-agnostic method:

class LogSink(Protocol):
    def write(self, record: LLMCallRecord) -> None: ...

A custom sink (a database, a metrics pipe, an in-memory buffer) is a one-method object that returns nothing; records (LLMCallRecord, a frozen dataclass) are handed to it for every call, and failures are swallowed so logging can never break a call. To send records somewhere other than local YAML — a database, an HTTP collector, structured stdout — implement write and register it:

import logging
from llmkit import LLMCallRecord, configure_llm_logging

logger = logging.getLogger("llm-calls")

class StructuredStdoutSink:
    def write(self, record: LLMCallRecord) -> None:
        logger.info(
            "llm_call",
            extra={
                "feature": record.feature,
                "label": record.label,
                "model": record.model,
                "provider": record.provider,
                "schema": record.schema,
                "duration_ms": record.duration_ms,
                "approximate_cost": record.approximate_cost,
                "error": record.error,
            },
        )

configure_llm_logging(StructuredStdoutSink())   # pass None to disable logging entirely

configure_llm_logging checks the object you hand it: anything that is neither None nor a LogSink — including the sink class where you meant an instance — raises TypeError there and then, rather than being installed and silently swallowing every subsequent call's log. The check is structural, so it tests that write exists, not that its signature is right (a type checker catches the rest at the call site). One consequence in your own tests: a bare MagicMock() doesn't register as a sink, because structural checks use static attribute lookup and see through no __getattr__ — stub with Mock(spec=LogSink), a small fake class, or configure_llm_logging(None).

The shipped LocalYamlLogSink additionally exposes the path it wrote via its own write_returning_path(record) -> Path | None method — that file detail stays off the shared LogSink contract, and it is what powers capture_llm_log_paths() internally. A sink that defines write_returning_path opts into path capture and must honor that return type: anything else is treated as a failed write (one warning, no captured path), so capture_llm_log_paths() only ever hands you real Paths.

An OpenTelemetry exporter (e.g. to Langfuse/Phoenix) is a natural future llmkit[otel] extra; the pluggable seam makes it a non-breaking addition.

Configuration

LLMClientConfig is flat and carries only what a call needs:

@dataclass(frozen=True)
class LLMClientConfig:
    provider: Provider               # OPENROUTER | OLLAMA | GOOGLE | ANTHROPIC | OPENAI | DEEPSEEK | BEDROCK | VERTEX
    model: str | None = None         # None -> the provider's own default model
    api_key: str | None = None       # bearer providers only; masked in repr; else the provider env var, else raises
    base_url: str | None = None      # endpoint: config, then env var, then a default (sent as api_base; see below); not accepted by Bedrock/Vertex
    reasoning_effort: ReasoningEffort | None = None  # "disable" | "low" | "medium" | "high", or any provider value
    aws_region_name: str | None = None   # AWS Bedrock region; not accepted by any other provider
    vertex_project: str | None = None    # Vertex AI GCP project; not accepted by any other provider
    vertex_location: str | None = None   # Vertex AI region (data residency); not accepted by any other provider
    gemini_structured_output: Literal["schema", "json"] = "schema"  # Gemini strategy; a non-default value is not accepted by any other provider

Populating a knob the selected provider does not read is rejected, not ignored. Each provider declares which provider-shaped fields it honours, and build_provider / make_provider raise a clear ValueError if the config carries any other populated one — an api_key for Bedrock/Vertex, a base_url for a fixed-endpoint provider, a non-default gemini_structured_output for a non-Gemini provider. Populate only the fields the active provider uses. (This closes a silent-misconfiguration footgun: a config generically filled from a settings object no longer looks like it pinned a credential or endpoint that was in fact dropped.)

api_key is masked in the config's repr. A set key renders as api_key=<redacted>, so a stray print(config), log line, or traceback never leaks the credential; its presence still shows for debugging.

Bearer-key resolution is explicit. The five key-authenticated providers resolve their key from api_key if set, else the provider's own environment variable, else they raise at construction naming the variable — no silent fallback to an ambient key:

OLLAMA needs no key; BEDROCK and VERTEX authenticate through their ambient AWS / Google credential chains (below), so none of the three accepts an api_key.

Endpoint resolution is explicit too. Every provider that accepts a base_url resolves its endpoint the same way — the configured base_url if set, else the first non-empty environment variable in the order below, else llmkit's own default — and sends the result on the wire as api_base. (GOOGLE is the one exception, deliberately: it owns no default, so with nothing configured it sends no api_base at all — see below.) So the endpoint is llmkit's decision, readable up front from the provider's completion_kwargs(), rather than something LiteLLM's internal chain resolves later. The resolution runs per call, not when the provider is constructed, so it reflects the environment the call actually goes out in: import llmkit does not import LiteLLM (that is deferred to the first call) and importing LiteLLM is what runs load_dotenv(), so a provider built once at startup with make_provider(...) and used later would otherwise answer from an environment the host's .env had not been read into yet.

The last two rows are governed differently from the six above them, and the section below says why and what it costs.

GOOGLE is the one provider with no llmkit default, and the absence is deliberate: its base is the one that cannot be a constant. The AI Studio base carries an API version that LiteLLM derives from the model — v1alpha for Gemini 3 and newer, v1beta otherwise — and it applies that only when no api_base is given; a base that is given is used verbatim. Measured against litellm 1.92.0 (2026-07-21): pinning https://generativelanguage.googleapis.com/v1beta sent gemini-3-pro-preview to /v1beta where it had gone to /v1alpha, and pinning the bare host dropped the version segment entirely — so any static default would silently change the wire shape for some models, which is the one thing this resolution promises not to do. Deriving the version inside llmkit would be endpoint computation, the gateway-shaped work the library deliberately does not do. So for Google: a configured base_url wins, else GEMINI_API_BASE, else no api_base is sent at all and LiteLLM picks the endpoint, and with it the model's API version. That last case has a real, scoped cost: with neither configured, Google's endpoint can still come from a source llmkit does not read — notably the litellm.api_base module global. Naming a base_url or setting GEMINI_API_BASE closes it.

The listed variables are still honoured — llmkit now reads them itself, in the order shown (LiteLLM's own measured precedence), so a host that points its endpoint with one of them is unaffected. What changed is the closure, and it covers OPENAI, ANTHROPIC, DEEPSEEK, and a GOOGLE that has a base_url or GEMINI_API_BASE: for those, sources llmkit does not read can no longer choose the endpoint — the litellm.api_base module global, a LiteLLM key-management backend serving one of these names, or an alias some future LiteLLM release adds. That is a closed fix rather than a blocklist chasing an unversioned dependency, and it is the same bargain api_key already strikes: make the ambient fallback explicit, documented, and tested rather than delete it. With nothing configured, the outbound request is byte-identical to before.

Two cases sit outside that closure. GOOGLE with neither base_url nor GEMINI_API_BASE set is one, per the paragraph above. OLLAMA is the other, and structurally so: both of its LiteLLM dispatch arms order the chain litellm.api_base or api_base or … — inverted relative to every other route — so for Ollama alone the module global outranks even the api_base llmkit sends (measured against litellm 1.92.0, 2026-07-21). That is accepted rather than fixed: it is a global a host has to set deliberately inside its own process, not something the ambient environment or a stray .env can reach, and closing it would mean reaching into a dependency's globals.

OPENROUTER and OLLAMA deliberately read no endpoint variable: both already named their endpoint in every configuration (a default that is a real URL, overridable via base_url), so honouring extra variables would widen the ambient surface for no gain.

BEDROCK and VERTEX do not own their endpoints

These two reject base_url outright, and llmkit sends them no api_base: their endpoints are derived rather than configured — Bedrock's from aws_region_name, Vertex's from vertex_location. llmkit names an endpoint where the endpoint is a constant it can measure once; where it is computed from an input llmkit does not own, llmkit declines to compute it and LiteLLM's own resolution stays in charge. (Google AI Studio with neither base_url nor GEMINI_API_BASE set is the third case in that same family, for the API-version reason above.) The alternative would be llmkit constructing regional URLs — the gateway-shaped work this library deliberately does not do, and for Vertex it would mean running Google credential resolution just to build a path, since the project id comes from ADC.

The cost is scoped, and it is sharper here than elsewhere, because these are the two providers whose region knob is a residency control. An ambient endpoint value overrides the region you pinned. Measured against litellm 1.92.0 (2026-07-23), with the region/location explicitly pinned in both arms:

BEDROCK, aws_region_name="eu-central-1"
  nothing set   -> https://bedrock-runtime.eu-central-1.amazonaws.com/model/us.anthropic.claude-haiku-4-5-20251001-v1%3A0/converse
  with the var  -> https://hijacked.invalid/model/us.anthropic.claude-haiku-4-5-20251001-v1%3A0/converse

VERTEX, vertex_location="europe-west4"
  nothing set   -> https://europe-west4-aiplatform.googleapis.com/v1/projects/<p>/locations/europe-west4/publishers/google/models/gemini-2.5-flash-lite:generateContent
  with the var  -> https://hijacked.invalid/v1:generateContent

Four things that matter if you pin residency:

• What travels. Vertex sends the ADC-minted Authorization: Bearer ya29.… to the overridden host; Bedrock hands it a live SigV4 signature over the real payload. This is credential exposure as much as region drift. Bedrock's SigV4 scope keeps following the region you pinned, so redirecting to a different AWS region fails loudly on signature; redirecting anywhere else succeeds silently.
• Vertex's surface is two sources, Bedrock's is one. For Vertex, LiteLLM resolves api_base or litellm.api_base or VERTEXAI_API_BASE on one line, and the in-process global outranks the variable — no environment hygiene reaches a global that any dependency or notebook cell can set. Bedrock's route never consults that global. (VERTEX_API_BASE, without the AI, is embeddings-only and does not affect completions.)
• Vertex replaces the whole URL rather than swapping the host, composing "{value}:{action}" — so a value carrying a path or trailing slash redirects silently, while a bare host fails loudly with Invalid port: 'generateContent' and sends nothing. Do not read a crash as the only failure mode.
• printenv is not a valid check. Importing LiteLLM runs load_dotenv(), and its search walks up from the installed package directory — so a .env sitting above your virtualenv is in force from any working directory. Both names are read through LiteLLM's get_secret, which also consults a configured key-management backend. An empty string is safe; a stray value is not.

Vertex has a second residency override with no environment variable involved. LiteLLM reroutes a pinned vertex_location to a model's first supported_regions entry when its shipped cost map lists regions that exclude yours, warning only on the verbose logger. llmkit's gemini-2.5-flash-lite default carries no such restriction today, but that is data in a dependency, not a guarantee — so treat vertex_location as pinning residency for models the installed LiteLLM does not constrain, and check a region-sensitive workload against the endpoint it actually reaches.

aws_region_name is the only AWS-shaped field, and it carries only the region. AWS Bedrock authenticates through the standard AWS credential chain (environment, shared config, or instance/role), so Bedrock secrets never pass through LLMClientConfig; leave the region None too and it resolves from the chain (AWS_REGION_NAME / AWS_REGION). Bedrock routing needs boto3 for request signing — install it with the opt-in extra:

pip install "omg-llmkit[bedrock]"

The default model is Claude Haiku 4.5 via its cross-region inference profile id (us.anthropic.claude-haiku-4-5-20251001-v1:0) — current Claude models on Bedrock are typically reached through inference profiles rather than plain on-demand ids. Pass a different profile- or partition-prefixed id as model (e.g. eu.anthropic.claude-...) when your account routes elsewhere.

vertex_project and vertex_location are the Vertex AI analog and are not accepted by any other provider. Vertex reaches the same Gemini models as the GOOGLE (AI Studio) provider, but through Google Cloud — and like Bedrock it carries no secret here: Google credentials resolve from Application Default Credentials (gcloud auth application-default login, GOOGLE_APPLICATION_CREDENTIALS, or a workload-identity / metadata-server token), never through LLMClientConfig. Leave vertex_project / vertex_location None to resolve them from the environment (VERTEXAI_PROJECT / VERTEXAI_LOCATION), with the location otherwise falling back to Google's default region. Vertex routing needs google-auth to mint its OAuth token — install it with the opt-in extra:

pip install "omg-llmkit[vertex]"

vertex_location is the data-processing residency control. It selects the endpoint where the request is processed, so a regional value (e.g. vertex_location="europe-west4") pins in-region processing; the "global" endpoint gives no residency guarantee. LiteLLM builds three endpoint shapes from it, not one: "global" yields https://aiplatform.googleapis.com, a multi-region geography ("us", "eu") yields https://aiplatform.<geo>.rep.googleapis.com, and any other value yields https://<location>-aiplatform.googleapis.com. llmkit does not construct that URL and does not send an api_base — see BEDROCK and VERTEX do not own their endpoints, which also covers the two ways a pinned location can be overridden. The default model is Gemini 2.5 Flash-Lite (parity with the AI Studio provider). As with AI Studio, Gemini 2.5 thinks by default — set reasoning_effort="disable" so a small max_tokens cap doesn't truncate structured output.

A residency region can constrain which model you may use. Gemini model availability is region-specific, so a region you pick for residency may not host every model — including the gemini-2.5-flash-lite default. A model that isn't deployed in your region fails with a Vertex 400 FAILED_PRECONDITION ("Precondition check failed."), which is an availability error, not an auth one. Pin a model the region actually serves (e.g. some regions offer gemini-2.5-flash but not -flash-lite). Check the Gemini-on-Vertex locations table for your region.

gemini_structured_output picks Gemini's structured-output strategy and is read only by the two Gemini providers (VERTEX, GOOGLE); a non-default value on any other provider is rejected (the default "schema" is always accepted). "schema" (the default) keeps Gemini's native JSON-schema constrained decoding (instructor.Mode.JSON_SCHEMA) with server-side schema enforcement — the pre-existing wire behavior, unchanged. "json" switches to Mode.JSON: the response is still server-side guaranteed to be JSON syntax, but the schema moves into the system prompt and is validated client-side (with instructor's single repair re-ask on a mismatch). Reach for "json" when a non-trivial schema drives runaway output loops: Gemini's constrained-decoding grammar mask is a repetition-loop trap — once the model starts looping, the mask blocks exactly the tokens that would break the pattern, so the call spins until max_tokens kills it (measured 67-83% first-attempt runaway under "schema" vs 0% under "json" on real prompts). The trade is giving up server-side schema enforcement for an occasional repair round-trip. An unrecognized value raises at provider construction rather than silently falling back.

configure_llm_client(lambda: LLMClientConfig(
    provider=Provider.VERTEX,
    gemini_structured_output="json",   # escape the constrained-decoding loop trap
))

Per-call model= overrides the default, so "strong/small/current" model roles are the host's concern — resolve them to a model string and pass it at the call site. The library has no opinion about roles.

reasoning_effort controls provider "thinking"/reasoning tokens, forwarded to LiteLLM. Leave it None (the default) for the provider's own behaviour — the outbound request is byte-identical to omitting it. Set it once (e.g. "disable") and every call inherits it; the call functions also take a reasoning_effort= override for a single call. This matters most for Gemini, whose thinking is on by default and spends reasoning tokens against max_tokens — reasoning_effort="disable" turns it off so a small max_tokens cap doesn't truncate structured output.

Register the config with configure_llm_client(source), where source is a zero-arg callable returning an LLMClientConfig (re-read on each provider construction, so it tracks live settings changes).

Constructing a provider for a per-call override

Most callers configure one provider once via configure_llm_client(...) and let every call pick it up. To send a single call through a different provider family, model, or credential, build a provider on the fly and pass it as the per-call provider= override. make_provider is the one-liner for that — it builds straight from raw credentials, with no LLMClientConfig and no module-level config source:

from llmkit import make_provider, structured_llm_call_sync, Provider

provider = make_provider(Provider.ANTHROPIC, api_key=anthropic_key)
result = structured_llm_call_sync(
    prompt,
    output_schema=MyModel,
    feature="summarize",
    provider=provider,
)

make_provider takes the same knobs as LLMClientConfig, and — like build_provider — rejects any that the selected provider does not read rather than silently dropping them: base_url for Bedrock/Vertex (whose endpoints derive from region/location), api_key for Ollama or Bedrock/Vertex (which authenticate via their ambient AWS/Google chains), aws_region_name for anything but Bedrock, and so on. Pass only the fields the provider uses. Leave model unset to inherit the provider's own default; the assembled LiteLLM id is always well-formed (e.g. anthropic/claude-sonnet-4-6).

A fully per-call host needs no global config at all. If you pass provider= on every call, you don't have to call configure_llm_client(...) — there is no global source to register, the call runs on the per-call provider alone, and the log records that provider as the effective one. The "configure once globally" and "provide per call" models are independent: use either, or mix them (a global default with per-call overrides where needed). A call that passes neither a per-call provider= nor a registered global source raises a clear RuntimeError telling you to configure one.

To build from a full config instead, use build_provider(config):

from llmkit import build_provider, LLMClientConfig, Provider

provider = build_provider(LLMClientConfig(provider=Provider.OPENAI, api_key=key))

LLMClientConfig.model is optional. When it is None (or empty), the provider falls back to its own built-in default model rather than emitting a broken "<prefix>/" id.

Naming: get_* reads, build_* / make_* construct

The accessor verbs are split by intent:

• build_provider(config) / make_provider(...) construct a provider — from a config, or from raw credentials.
• describe_llm(config) (importable from llmkit.providers) and get_rate_limit_config() read effective state — a snapshot for display/telemetry; they construct nothing you keep.

describe_llm replaces the old get_llm_config, and build_provider replaces get_provider; both old names are gone from the public surface.

OpenRouter: schema-honoring routing

OpenRouter is a router — it forwards your request to one of several serving providers behind each model. There's a sharp edge for structured output: structured_outputs is a model-level capability, but the strict response_format is actually enforced by the serving endpoint the request lands on. A model can advertise the capability while one of its endpoints quietly ignores the schema and returns free-form JSON — which then surfaces only as a confusing downstream validation failure, with nothing pointing at routing as the cause.

OpenRouterProvider defends against this by default: it sets OpenRouter's provider.require_parameters routing preference, so a request only lands on a serving endpoint that honors every parameter sent — including the structured response_format. The trade-off is that restricting routing to capable endpoints can in principle reduce availability or shift cost. To opt out (and accept the silent-free-form-JSON risk), construct the provider directly:

from llmkit import structured_llm_call
from llmkit.providers import OpenRouterProvider

provider = OpenRouterProvider(api_key="sk-or-...", require_parameters=False)
result = await structured_llm_call(prompt, MySchema, feature="x", provider=provider)

Routing stays on for the config-driven path (configure_llm_client / build_provider); the direct constructor above is the way to turn it off.

Retries

Two retry layers, kept deliberately separate:

• Transient-provider retries, on by default. Every call function (structured_llm_call, structured_llm_call_sync, text_llm_call, text_llm_call_sync, text_llm_call_stream) retries transient provider errors on its own — you don't wrap anything. The recoverable set splits into two budgets the policy counts separately:

  • Transport errors (LLM_TRANSPORT_ERRORS: 429 / 503 / 5xx, network/timeout) get the full max_attempts budget — three attempts by default — since a retry on a fresh connection routinely succeeds.
  • Schema-validation errors (LLM_SCHEMA_ERRORS: pydantic ValidationError, instructor InstructorRetryException) get the lower validation_max_attempts budget — two attempts (one retry) by default — so a transiently-malformed JSON response is still recovered, but a deterministically-wrong schema can't burn the full transport budget on doomed re-asks. (instructor wraps transport failures in InstructorRetryException too; the retry layer unwraps it, so a wrapped 429/5xx/network error still gets the full transport budget, not this lower one — and a wrapped permanent error such as a 401/400/403 fails fast after a single attempt, never charged to either budget.)
  • Output-limit truncations (LLM_OUTPUT_LIMIT_ERRORS: llmkit's own OutputLimitError, raised when a structured completion is cut off by the output-token limit, finish_reason='length') get zero budget — never retried: a re-ask with an identical token budget can only truncate again (the motivating production failure was a degenerate repetition loop that burned to the provider's 65k-token ceiling on the original ask and on every blind re-ask, turning a seconds-long call into minutes of doomed generation). The error carries model / max_tokens / completion_tokens, so the fix is legible from the error alone: completion_tokens at a cap you set means raise the cap; a huge count under no cap means the prompt induces runaway output. A caller that genuinely wants the resample can opt back in by listing OutputLimitError explicitly in retry_on.

  LLM_RECOVERABLE_ERRORS remains the documented single catch-set — now the union of four subsets: the three above plus LLM_BACKPRESSURE_ERRORS (llmkit's own fail-fast CircuitOpenError, raised by the opt-in circuit breaker). Keep using LLM_RECOVERABLE_ERRORS in except clauses; the split only changes how the retry layer budgets them — and the backpressure/output-limit subsets are deliberately never retried (re-asking a known-open circuit or a same-budget truncation is doomed by construction). One footnote on the 503 case: so that import llmkit doesn't pay LiteLLM's multi-second import cost, litellm's own 503 class (litellm.exceptions.ServiceUnavailableError) is never imported eagerly — instead, llmkit re-raises every litellm-native 503 at its transport boundary as llmkit's own ServiceUnavailableError, a plain member of LLM_TRANSPORT_ERRORS, so except LLM_RECOVERABLE_ERRORS: genuinely catches it. It carries provider, model, status_code (always 503), and the original response (so a server Retry-After stays honoured), with the original litellm error on __cause__. If you previously caught litellm.exceptions.ServiceUnavailableError directly around llmkit calls, catch llmkit.ServiceUnavailableError (or the documented tuple) instead; the raw litellm class still classifies as transport in isinstance checks — e.g. your own litellm call wrapped in with_retries — via a lazy stand-in resolved once litellm is loaded. Both budgets use bounded full-jitter backoff: the sleep before retry n is a random delay in [0, min(backoff_base_seconds * 2**(n-1), max_backoff_seconds)], with the per-sleep cap (max_backoff_seconds) defaulting to 30s so a large attempt budget can't grow the worst-case sleep unboundedly. Retry-After is honoured first: when a retried provider error carries a Retry-After (a header — delta-seconds, retry-after-ms, or an HTTP-date — or the SDK's numeric attribute), the backoff waits that duration instead of the computed exponential, capped at RetryPolicy.retry_after_cap (default 60s) so a hostile value can't wedge a call. It is read from the unwrapped provider error, so a structured call honours it too, and it is honoured even when backoff_base_seconds is 0 (a server directive, not opt-in backoff); absent a header, the exponential is used unchanged. Programming errors (e.g. TypeError) are outside the recoverable set and propagate immediately, never retried. Each attempt is its own logged call, so data/llm-logs/ shows one record per attempt.

  Tune or opt out per call with the retry= argument:

  from llmkit import structured_llm_call, RetryPolicy, NO_RETRY
  
  # Opt this one call out of automatic retries (e.g. latency-sensitive):
  result = await structured_llm_call(
      prompt="Summarize the attached report.",
      output_schema=Summary,
      feature="reports",
      label="exec_summary",
      retry=NO_RETRY,
  )
  
  # Or tune the budget / backoff for this call:
  result = await structured_llm_call(
      prompt="Summarize the attached report.",
      output_schema=Summary,
      feature="reports",
      label="exec_summary",
      retry=RetryPolicy(max_attempts=5, backoff_base_seconds=1.0),
  )
  

  Streaming caveat: text_llm_call_stream can only retry a transient failure that happens before the first chunk reaches the caller. Once any chunk has been yielded, a mid-stream error propagates unretried — a partially-consumed stream can't be safely restarted.

  with_retries() (imported from llmkit.retry; see retry.py) remains the explicit, composable advanced path for wrapping any awaitable — useful when you want to retry a unit of work that isn't a single call function. The attempt count is max_attempts (total attempts including the first, N not 1+N); the previously-deprecated max_retries alias has been removed outright, so passing it now raises TypeError. Wrap a retry_progress_callback(...) scope around the work to observe per-attempt failures (e.g. for a progress UI):

  from llmkit.retry import with_retries
  from llmkit import LLM_TRANSPORT_ERRORS
  
  result = await with_retries(
      lambda: do_some_work(),
      max_attempts=3,
      backoff_base_seconds=0.5,
      retry_on=LLM_TRANSPORT_ERRORS,
  )
  

  A RetryProgressCallback is invoked once per non-final failed attempt with keyword arguments label, attempt, max_attempts, and error — the callback keyword is max_attempts (it was previously max_retries; rename it):

  def on_retry(*, label: str, attempt: int, max_attempts: int, error: BaseException) -> None:
      print(f"{label}: attempt {attempt}/{max_attempts} failed: {error}")
  

  Don't double-wrap the call functions. They already retry internally, so with_retries(structured_llm_call, ...) would otherwise multiply the budgets (the 3 × 3 = 9 trap). with_retries guards against this — it detects an active llmkit retry loop owned by the current asyncio task and collapses the inner layer to a single pass (warning once), so the budgets don't multiply. The task scoping bounds the guard to the pattern it warns about: a nested loop awaited inline in the same task, whose failure really does propagate out to the outer loop. A distinct llmkit call that merely inherited the scope across a task boundary keeps its full retry budget — an on_result hook calling structured_llm_call_sync / text_llm_call_sync (the sync bridge drives the coroutine as a new task on llmkit's persistent loop), or a call spawned with asyncio.create_task from inside an attempt. To drive retries entirely from your own wrapper instead, opt the inner call out with retry=NO_RETRY.

• instructor's own in-call schema repair re-asks the model to fix malformed JSON within a single call, before any ValidationError/InstructorRetryException reaches the retry layer. llmkit pins instructor's budget to two in-call attempts (a per-call tenacity AsyncRetrying stopping after 2 — instructor counts total attempts, so that is exactly one repair re-ask) — and it is not a caller-facing knob. The re-ask fires for a genuine parse failure only: malformed JSON, a Pydantic ValidationError (a failing async field validator included), or instructor's own ResponseParsingError (e.g. a blocked Gemini Mode.JSON response). Every other failure is declined by the in-call loop, so it costs exactly one provider request per attempt: a transport failure (429/5xx/network) leaves the rate-limiter slot immediately and is retried by the cross-call layer above, with backoff and Retry-After honoured, rather than being re-sent inside the same slot with neither; a permanent failure (401/400/403) fails fast with no in-call duplicate; and a completion truncated by the output-token limit is never re-asked (the re-ask would run on the identical budget and can only truncate again) and surfaces immediately as OutputLimitError. This stays separate from the cross-call retry layer above: instructor repairs within one attempt; the policy's validation_max_attempts (default 2) governs how many fresh attempts a persistent schema failure earns. The two budgets are never conflated, so attempts aren't double-counted.

Re-rolling on a semantically-bad result

A response can pass the schema and still be wrong — an empty register, a citation that doesn't resolve, a total that doesn't reconcile. Rather than hand-rolling an LLM-then-validate-then-re-roll loop around the call, pass an on_result hook: it's called with each attempt's result, and raising ResultValidationError from it rejects that result and re-rolls the call.

from llmkit import structured_llm_call, ResultValidationError

def _must_have_findings(report: Report) -> None:
    if not report.findings:
        raise ResultValidationError("empty report — re-roll")

result = await structured_llm_call(
    prompt, Report, feature="reports", on_result=_must_have_findings,
)

The re-roll is charged against the validation budget (RetryPolicy.validation_max_attempts, default 2) — the same budget a schema failure uses, and for the same reason: a deterministically-bad result shouldn't burn the full transport budget on doomed re-asks. When the budget is exhausted the last ResultValidationError propagates. Each attempt — including a rejected one — is its own logged call, so data/llm-logs/ shows the rejected response alongside the error.

on_result is available on structured_llm_call and text_llm_call, and on both sync wrappers (structured_llm_call_sync, text_llm_call_sync); the text-path hooks receive the response text. It is not part of LLMCallOptions — like feature, it stays a conscious per-call choice.

Development

uv sync
uv run ruff check . && uv run ruff format --check .
uv run basedpyright          # recommended tier; clean with no baseline
uv run pytest

Status & support

llmkit is a small, opinionated, best-effort project, extracted from a real application and maintained in the open. It is used in production by its author but carries no support SLA. Bug reports and focused pull requests are welcome — see CONTRIBUTING.md. For security issues, see SECURITY.md.

License

MIT — see LICENSE.