runlatent — ingest finished LLM calls into Latent's scoring API

pip install runlatent

A Python SDK that captures every finished call your app makes through the OpenAI, Anthropic, Google GenAI, Ollama, Groq, Together, Fireworks, Mistral, Cohere, Azure AI Inference, Hugging Face or boto3 Bedrock Python SDK (and every OpenAI-compatible endpoint through the openai client: the provider matrix with the base_url recipe per host is docs/sdk-providers.md) and posts the (prompt, answer) pair to Latent's scoring API, which is the sidecar's POST /score contract (latent_sidecar/README.md). The app's call is never slower by a network round trip, never sees a different return value and never gets an exception from here. Step 1 of the self-serve path: no gateway, no log tail, one line at startup.

your app ──openai / anthropic / google-genai / ollama SDK──▶ the provider
   │ (after the SDK returned, fire-and-forget, a daemon thread)
   └──POST <LATENT_API_URL>/score {request_id, messages, output, model, finish_reason}──▶ Latent

Install

pip install runlatent (the runlatent distribution on PyPI, built from runlatent/ in this repository; from the repository, pip install -e .). It has no runtime dependencies: nothing from numpy, torch or the core packages reaches your process. The provider SDKs are yours: whichever of openai, anthropic, google-genai, ollama, groq, together, fireworks-ai, mistralai, cohere, azure-ai-inference, huggingface_hub, boto3 is importable gets patched, the others are skipped (docs/sdk-providers.md lists what each one covers and the hosts behind the openai client).

One-line setup

import runlatent
runlatent.auto_instrument()        # reads LATENT_API_URL and LATENT_API_KEY

Call it once, before the clients are built or after, it does not matter. Supported SDKs: OpenAI, Anthropic, Gemini, Ollama (and the rest of the list above). The patch is on the SDK classes (Completions.create, AsyncCompletions.create, Responses.create, AsyncResponses.create; Messages.create / .stream and the async pair; Models.generate_content / .generate_content_stream and AsyncModels; ollama's Client.chat / .generate and AsyncClient, plus the module-level ollama.chat / ollama.generate, which are bound to the module's own client at import), so every client in the process is covered, including ones a framework builds for you. Keyword arguments:

argument	default	meaning
openai, anthropic, google_genai, ollama, groq, together, fireworks, mistralai, cohere, azure_ai_inference, huggingface_hub, bedrock	True	which SDKs to patch (docs/sdk-providers.md)
api_url	$LATENT_API_URL	the API base; the SDK posts to <api_url>/score
api_key	$LATENT_API_KEY	sent as Authorization: Bearer <key>
redact	None	callable(body) -> body or None, run on every pair before it is queued (below)
timeout	5.0	seconds per POST
max_queue	10000	pairs waiting for the poster thread; a full queue drops, counted

Idempotent: a second call patches nothing twice, and with the same destination it keeps the same poster and its queue; a call with a different api_url / api_key / timeout / max_queue swaps the poster at once and drains the old one on its own thread (its final counters stay in stats()), never on the caller and never while the app's calls wait. runlatent.uninstrument() restores every original (classes and wrapped instances). With no URL anywhere, auto_instrument() logs one warning, patches nothing and returns stats() with enabled: False.

Any OpenAI-compatible endpoint. The patch is on the openai client, not on OpenAI's servers, so any OpenAI-compatible server (Ollama's /v1, vLLM's OpenAI server, LM Studio, a Hugging Face Inference Endpoint or TGI, Together, Fireworks and the like) is captured the moment the app talks to it through the openai client with base_url pointed at it, chat and streaming alike, with the model name the app passed:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")   # Ollama's /v1; vLLM: http://<host>:8000/v1
client.chat.completions.create(model="llama3.2", messages=msgs)          # captured like any OpenAI call

Ollama's native client (pip install ollama) is captured as is, no URL recipe: ollama.chat(...), ollama.generate(...), ollama.Client().chat / .generate, ollama.AsyncClient, stream=True on each (the chunks are assembled). A generate call posts its system as the system turn and the prompt as one user turn (a fill-in-the-middle suffix follows it under a [suffix] marker, a prior-turn context of token ids is a counted [context omitted] part); the finish is done_reason, usage is prompt_eval_count / eval_count, and since Ollama returns no response id the pair's request_id is the SDK's own (client.response_id is null). embed and the model management calls are not answers and are not patched.

import ollama
ollama.chat(model="llama3.2", messages=msgs)                   # captured; so is ollama.Client().chat(...)

Anything else (in-process transformers generation, Replicate, SageMaker invoke_endpoint, a client the SDK does not patch) uses runlatent.check directly after each generation: one POST, synchronous, the verdict in hand (below). Or serve the model through vLLM with Latent's in-band plugin and skip the SDK altogether:

answer = tokenizer.decode(model.generate(**inputs)[0], skip_special_tokens=True)
r = runlatent.check(prompt, answer, context=docs, model="meta-llama/Llama-3.1-8B-Instruct")
if r.flagged: answer = SAFE_FALLBACK

Check before you deliver: runlatent.check

auto_instrument and the wrappers observe: the pair is posted after the provider returned, on a daemon thread, and the read never blocks or fails your call. To hold, replace or re-route a flagged answer before your user sees it, read it synchronously first. On hosted plans POST /score answers the verdict in tens of milliseconds (the hosted service: docs/hosted-score.md), so the whole pattern is two lines:

r = runlatent.check(prompt, answer, context=docs)
if r.flagged: answer = SAFE_FALLBACK

check is the explicit way to hold before delivery on hosted plans; in front of your own vLLM the gateway does it automatically from the in-band event. auto_instrument holds, replaces or re-routes only under the console policy (its default enforce="policy", the section "Act from the console" below); with enforce=False it never changes a response.

runlatent.check(prompt, answer, *, context=None, system=None, messages=None, model=None,
                timeout=2.0, request_id=None, raise_on_error=False, detail=False) -> CheckResult
await runlatent.acheck(...)      # the same read, awaited (run off the event loop)

argument	meaning
prompt	the user's question (one user turn)
answer	the text about to be delivered; empty -> verdict: "unscored", nothing posted
context	the retrieved sources; rendered before the question as Knowledge: <context>\n<prompt>, the shape the probes are calibrated on. Pass it: an answer with no source (no context, no Knowledge: line, no system) is stored and shown but gets no grounding verdict and is never flagged (verdict: "unscored", the server's unscored_reason: "no_source")
system	the system prompt, a string or Anthropic's block list, posted as a leading system turn (flattened as the capture flattens it)
messages	the conversation as the provider saw it ([{role, content}, ...]), instead of prompt / context
model	the generating model's name, for the console
timeout	seconds for the whole exchange (default 2.0); no reply in time is a refusal, never an exception
request_id	the console row's id (the provider's response id when you have one; else a uuid)
raise_on_error	True raises runlatent.CheckError (with .result) on a refusal instead of returning it
session_id / user_id	the call or conversation this turn belongs to and its end user (your own opaque ids, never personal data); else the ambient runlatent.session(...) block, else the fingerprint inferred from messages ("Group the turns of a call", below)
detail	True asks the same read for its metadata in the same reply (detail: true in the body; no second pass): r.sentences, r.suite / r.suite_fired, r.threshold and the raw block r.read (below, "Console-free use")

It uses the destination and key auto_instrument() / configure() / the environment set (LATENT_API_URL, LATENT_API_KEY), posts the same body auto_instrument posts (client.api: "check") through the same redact hook, and the route stores the pair in your console like any other scored answer. With neither a URL nor a key configured it raises runlatent.ConfigurationError at once, whatever raise_on_error says: nothing can be checked and you need to know.

CheckResult is a small frozen dataclass:

field	meaning
risk	the probe's probability that the answer is unreliable (None when not scored)
verdict	pass / escalate / ood from the reader; unscored (an empty answer, or redact returned None); unknown (the read did not happen, or came back unreadable)
request_id	the console row (/events/<request_id>)
ood_distance	the OOD guard's distance, when the reader sent one
scored	True exactly when the server answered 2xx with a risk and a verdict the SDK knows
status	the HTTP status; 0 when no reply came back (timeout, connection error, local skip)
detail, resume	the server's detail and how-to-resume sentence on a refusal (limit_reached, rate_limited, unauthorized: ...), or the transport error
retry_after_s	the server's Retry-After on a 429 / 503
error	the exception class on a transport failure (TimeoutError, ConnectionRefusedError)
.flagged	verdict in ("escalate", "ood"): do not deliver as is
.passed	scored and pass
.refused	the read did not happen or was unreadable (verdict == "unknown")
read	with detail=True: the server's raw detail block (docs/hosted-score.md, "The detail block"), the turn's session included when you passed session_id / user_id or an ambient runlatent.session(...); else None
sentences	with detail=True: the sentence read, a tuple of frozen Sentence(start, end, text, risk) in answer order (r.read["sentences_status"] says why it is what it is: "ok", "absent" = no sentence probe on the reader, "failed" = the configured read raised, with r.read["sentences_reason"], "skipped" = nothing to read; a failed read is marked, never reported as absent) (answer[start:end] == text when r.read["text_source"] is "output", which it is for a text-only check; "decoded" marks offsets into the reader's decoded answer when posted token ids did not round-trip; risk the sentence probe's probability that the sentence is unsupported); None when no sentence probe is attached to the reader, or without detail
suite, suite_fired	with detail=True: the probe heads' verdicts by name, frozen SuiteHead(risk, threshold, over, enabled) (grounding, arithmetic, a head calibrated on your traffic), and the names that fired; None / () when the artifact carries no heads, or without detail
.threshold	with detail=True: the cut the verdict was judged against; else None

A refusal never raises by default. A 401 / 403 / 429 / 5xx, a timeout and a connection error all return scored=False with the status and the server's words, so the fail-open / fail-closed choice is yours and explicit:

r = runlatent.check(prompt, answer, context=docs)
if not r.scored:                # the read did not happen: r.status, r.detail, r.resume say why
    deliver(answer)             # fail open: deliver unchecked (and alert on r.detail), or
    # answer = SAFE_FALLBACK    # fail closed: hold
elif r.flagged:
    answer = SAFE_FALLBACK

if r.passed is the fail-closed test in one word (a refusal is not a pass); if not r.flagged is the fail-open one. bool(r) is r.passed, so if runlatent.check(...) holds a refusal AND an empty answer (unscored); the properties say which. raise_on_error=True turns a refusal into runlatent.CheckError for code that prefers an exception. A 2xx whose verdict is not one of pass / escalate / ood is a refusal too (scored=False, verdict unknown, error ValueError): a reply the SDK cannot read never delivers as "not flagged". The only other exceptions are ConfigurationError (no URL or no key) and ValueError for a call that cannot mean anything (messages= together with prompt / context; a timeout that is not positive).

One scored answer, not two. With auto_instrument on, the pair the instrumentation just captured is scored once when the check can be paired with it. The pairing is by request_id first (pass the provider's response id, request_id=resp.id: exact), then by the conversation (messages= as the provider saw it, plus the answer, whitespace normalized; with Anthropic pass system= too), never by the answer text alone: the grounding verdict depends on the prompt, so "Paris" checked for "Capital of Germany?" must not inherit the pass of "Paris" captured for "Capital of France?". Paired, check adopts the captured read's reply and its request_id (stats()["check_adopted"]); a captured read still queued behind a slow one after half the timeout is claimed and posted by check under the same id, the poster skipping its copy (skipped_checked); one the poster is already sending, or retrying after a 429 / 5xx, is waited for to the end of the timeout and adopted (check_waited_in_flight), never posted a second time; a read that outlasts the timeout is a refusal (status 0, error TimeoutError, "still in flight") and the poster's read lands in the console on its own. A check that runs before the capture fires (Anthropic's .stream() manager: the app has the text inside the with) makes that capture skip when it carries the same request_id (get_final_message().id) or the same messages=. One capture pairs with one check, within 60 s, in a set bounded to 1,000 pairs.

What is not paired is scored twice, two console rows and two scored answers on the bill: check(prompt, answer, context=docs) with no request_id next to auto_instrument (it renders one user turn, Knowledge: <context>\n<prompt>, which equals the provider's messages only when that is exactly what you sent; otherwise only the id pairs them: pass resp.id), and a stream manager's check that carries neither the id nor the messages. A paired answer appears once in the console and once on the bill.

Console-free use: the risk score and its metadata in one reply

A service that wants no console, just "send the full request and get back a risk score plus its metadata and classifications", asks for detail. The reply is the same read: nothing is computed for the block, it is what the reader already had on the event (the cut, the OOD signal, the sentence read, the probe heads, the finish, the token count, who read it), serialized. One POST, no second pass.

r = runlatent.check(prompt, answer, context=docs, detail=True)
print(r.risk, r.verdict, r.threshold)             # 0.71 escalate 0.5
for s in r.sentences or ():                       # the sentence read, in answer order
    print(f"{s.risk:.2f}  {s.text}")              # 0.04  Paris is the capital of France.
                                                  # 0.91  It has two million people.
for name, head in (r.suite or {}).items():        # the classifications: grounding / arithmetic / your heads
    print(name, head.risk, head.over, head.enabled)
print(r.suite_fired)                              # ('grounding',)

The same with curl, against the hosted API (docs/hosted-score.md, "The detail block"; ?detail=1 for a body you do not want to edit, the body's own detail winning when both are given):

curl -s https://api.runlatent.ai/score -H "Authorization: Bearer $LATENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"request_id": "req-1", "messages": [{"role": "user", "content": "Knowledge: ...\nCapital of France?"}],
       "output": "Paris is the capital of France. It has two million people.", "detail": true}'

r.sentences[i].text is your own answer sliced at the read's sentence spans (echoed back, never stored text), so the block is available on every plan. Not in the block, by design: the judge's rationale (the console's request page), the per-token trace (its own keys on the reply, token_risk and friends) and the explain record (POST /explain on the sidecar, docs/explain.md). Without detail=True the reply and the result are exactly what they were.

A detail=True check never adopts a captured read (the instrumentation posts without detail, so the adopted reply would carry none): when the poster has not started on the pair the check claims it and the one post carries detail (skipped_checked); when the poster already sent it the check posts its own read under the same request_id (stats()["check_detail_reposted"]), one console row and the answer scored and billed twice. Pass request_id=resp.id and call check before the capture fires to keep it at one.

Act from the console: the policy enforced in the SDK (enforce="policy")

check() is for code you write. The console's Act automatically needs no code: auto_instrument() (and the wrappers) default to enforce="policy", which applies the project's policy to the answers your app is about to use.

runlatent.auto_instrument()                 # enforce="policy": the console decides (the default)
runlatent.auto_instrument(enforce=False)    # never change a response; the console shows "enforcement disabled in the SDK"

How it works: the SDK polls your project's policy, GET <LATENT_API_URL>/policy with your key, every 60 s (ETag, so an unchanged document costs one 304; a failed poll keeps the last document and backs off up to 5 min; a 401 drops it and is logged once). The poller starts at auto_instrument() time and no call ever waits for it: until the first document has arrived the SDK observes only (those pairs are posted with enforce.reason: policy_not_loaded), and enforcement begins once the document is in and says to act.

Per model (SDK 0.3.6). The console's policy is one default document plus one partial override per model, and the SDK acts on the composed policy of the model of each call: beside the default it polls GET /policy?model=<id> (the default with that model's override on top) for every model with an override, read from the default document's models map and published together with the default (a tick reads the default and every override copy it names, then swaps them in at once), so no call ever sees a default whose overrides are still pending: the override is in force from the first call that names its model, at the first load and after every move of the default; and for every model your calls name (registered on first sight and fetched on the poller's thread at once; that first call is decided by the default, never delayed), each with its own ETag and the same 60 s refresh (the per-model reads of a tick run on a small pool, so a slow control plane never holds the default's refresh). A composed copy is in force only while it is at the default's version or newer (the control plane numbers both views alike): a copy behind the default is not, and the default decides until the copy's next read, unconditional, on the same tick. The one case a call waits: a model the cache knows has an override whose copy is NOT in force (its read failed, or the default moved and the copy's read failed) waits up to the policy's wait budget (response.wait_ms, once per window) for the copy, and past the budget is decided by the default and marked pending_override: true on the decision block and the posted pair (counted enforce_override_pending), never silently. The decision block, runlatent.last_decision() and stats()["enforce"] carry policy_source (default, or model:<id>), the posted pair's enforce.policy_source the same, and stats()["enforce"]["policy"]["models"] lists the composed documents in the cache (version, pending, stale, in_force, override, pending_override, fetched / 304 / failures; models_dropped is the count of distinct models past the 64 kept, decided on the default; an override always outranks a lazily registered model for a place). An async call is enforced on the model the provider's RESPONSE names (an alias resolved to a model with its own override is decided by that override, a route re-issued), as a sync call is. A Hold saved on a model's override alone therefore holds that model's calls and no other's; a Hold on the default holds every model unless its override says otherwise. (0.3.5 read the default document alone.) While the policy's Automatic response is none (or the in-band plugin's early stop) nothing changes: the call path is the observe path, byte for byte (no read, no wait; the pair is posted in the background as before). While it is abstain, regenerate, route or judge, every non-streaming, single-sample captured call is scored synchronously before the response is returned to your app: one POST /score within the policy's wait budget (response.wait_ms, default 500 ms, the same default as the gateway's), then the decision the gateway takes (the same table, runlatent/_decision.py, a verbatim copy of latent_gateway/decision.py) and the action in place:

response.action	what your app gets
abstain	each choice's content replaced with the policy's fallback_text (ids, usage and the model kept; tool calls, function calls and logprobs dropped; a tool-call finish becomes stop); response.latent.action == "abstain"
regenerate	the same call re-issued once (stream off) and the NEW answer delivered: latent.verdict == "regenerated", latent.risk null, the withheld answer's numbers under original_risk / original_verdict, regenerated_id; the new answer is posted to the console as its own row (enforce.reason: reissued_answer, never re-read before delivery). A re-issue that fails delivers the ORIGINAL flagged answer as regenerate_failed (the gateway's rule), unless the policy pins response.fail_closed: true, which holds it (the fallback text, action: hold)
route	the same call re-issued to response.route_model on the SAME provider (latent.verdict == "routed", routed_to); a route_model of another provider (anthropic:claude-... under an OpenAI client) is route_not_possible_in_sdk, no model is route_unconfigured: the answer is delivered unchanged with its verdict. A re-issue that fails: as regenerate, route_failed, held under fail_closed: true
judge	delivered unchanged; the pair is queued for the console's explain judge through the report below

The scope is complete, single-choice answers: a streamed answer and an n > 1 call are delivered as the model produced them, with the verdict, never changed (the console's Policy page says so beside Act automatically, and, when every recent answer of yours was streamed, that the action is not applied to your traffic). | below the flag cut | delivered unchanged, latent.action == "none", reason == "below_flag_cut" |

Providers covered. OpenAI chat completions and Responses, Anthropic Messages, Google GenAI generate_content, Ollama chat and generate (the native client): each provider's response shape is rewritten in place (OpenAI: every choice's content, tool calls / function calls / logprobs dropped, a tool-call finish becomes stop; Anthropic: the first text block kept with the replacement, tool-use and thinking blocks dropped, stop_reason: tool_use becomes end_turn; GenAI: the first candidate's first text part, the other parts and candidates dropped; Ollama: message.content / response, the message's tool_calls and the thinking text dropped, model, done_reason, the eval counts and a generate call's context kept). A route on Ollama re-issues the call to the policy's model on the same client (llama3.1:70b is a model tag, ollama:llama3.1:70b the same target spelled with its provider; openai:gpt-5 from an Ollama call is route_not_possible_in_sdk). A response shape the SDK cannot rewrite is delivered as the model produced it with latent.reason ending in +replace_failed (counted enforce_replace_failed), and its action report says so (released: true, replaced: false, SDK 0.3.7), so the console's "held" count never includes it; the SDK never substitutes a shape of its own.

Latency. Only the calls the policy acts on pay the synchronous read: one /score round trip, bounded by the policy's wait budget (response.wait_ms, 500 ms default); the reader's own time depends on the deployment and is not quoted here. A regenerate or a route adds one model call. Streams never wait (a stream is never enforced; its capture says stream_not_enforced:<action>), n > 1 calls never wait (n_not_enforced:<action>), and with enforce=False nothing waits.

Failure semantics are the policy's. No read within the budget (a timeout, a 5xx, a 429, an empty answer): under a policy that withholds flagged answers (abstain) the answer is HELD, the fallback text delivered with latent.action == "hold", reason == "unmatched_timeout" (fail-closed, the gateway's auto rule); under regenerate / route / judge it is released unchanged with latent.verdict == "unknown" (fail-open); response.fail_closed: true | false in the policy pins either. A read that did not come back is re-posted in the background (hosted ingest is idempotent on request_id, so a row the slow reader stored anyway dedupes) and the decision report waits for that post, retrying a 404 (the row not there yet) or an unreachable control plane after 1, 3 and 8 s, so the console shows the hold or the release on the row it was taken on (enforce.reason: read_not_returned on the re-posted pair). A policy route that refuses the key (401) means nothing is enforced until the key is accepted; a destination with no policy route at all (a sidecar: 404) is said once and enforces nothing. Your redact hook returning None for a pair the policy would have read is not silent either: under a withholding policy (abstain) the answer is held (latent.reason: redact_none, the fallback text), under any other it is delivered with latent.reason: redact_none; nothing is posted or reported for it, as the hook asked. The app's calls are never failed by any of this: every failure on the enforcement path (the read, the replacement, the report, the poller, a re-issue that raises) is swallowed and counted (stats()["errors"]) and the provider's own response object is returned (pinned per failure path in tests/test_sdk_enforce.py).

What travels. The decision is attached to the response as latent when the SDK's model takes an extra attribute (openai's and anthropic's pydantic models do: response.latent / response.model_extra["latent"]; google-genai's GenerateContentResponse and ollama's ChatResponse / GenerateResponse forbid extras, so there runlatent.last_decision() is the way to read it, and the pair counts enforce_block_not_attached), and is always runlatent.last_decision(): {risk, verdict, action, reason, policy_version, tier, cut_basis, flag_cut, enforced_by: "sdk", request_id, wait_ms, ...}. Every posted pair carries enforce: {by: "sdk", enabled, policy_version, applied, reason} so the console knows the SDK is enforcing (its Policy page then offers Act automatically) or that you turned it off; each decision is reported to POST <base>/score/action (fire-and-forget, bounded) and shows on the request's row as action_applied. LATENT_SDK_ENFORCE=off is the environment's spelling of enforce=False; an explicit enforce= argument wins. With enforcement off the policy is not polled at all.

Per-client wrappers

The same capture on one client object, nothing else in the process touched:

client = runlatent.wrap_openai(OpenAI())          # or AsyncOpenAI(); any OpenAI-compatible base_url
client = runlatent.wrap_anthropic(Anthropic())    # or AsyncAnthropic(), AnthropicBedrock(), AnthropicVertex()
client = runlatent.wrap_genai(genai.Client(...))  # sync and .aio, Gemini API or vertexai=True
client = runlatent.wrap_ollama(ollama.Client())   # or ollama.AsyncClient(); chat and generate, streams included
client = runlatent.wrap_groq(Groq())              # likewise wrap_together, wrap_fireworks, wrap_mistral,
                                                  # wrap_cohere, wrap_azure_inference, wrap_huggingface,
                                                  # wrap_bedrock(boto3.client("bedrock-runtime"))

Each returns the client. Wrapping twice is a no-op; a wrapped instance under a patched class captures once, not twice.

Environment

variable	meaning
LATENT_API_URL	the API base (the sidecar's address when you run it yourself: http://sidecar:8090)
LATENT_API_KEY	the bearer (the sidecar's LATENT_SIDECAR_TOKEN)
LATENT_SIDECAR_URL, LATENT_SIDECAR_TOKEN	accepted as fallbacks when the two above are unset. A sidecar has no /policy route: the policy poll answers 404, said once at INFO, and nothing is enforced (the sidecar path is observe-only)
LATENT_SDK_EXIT_FLUSH_S	seconds the atexit handler waits for queued pairs at a normal interpreter exit (default 2.0; 0 disables, see Limits)
LATENT_SDK_ENFORCE	off / 0 / false = never apply the console policy to a response (auto_instrument(enforce=False)); unset = policy ("Act from the console"); an explicit enforce= argument wins

What is posted

One JSON body per finished call, the sidecar's /score contract plus one extra block:

{"request_id": "chatcmpl-abc123",
 "messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}],
 "output": "the full answer text",
 "model": "gpt-4.1", "finish_reason": "stop",
 "client": {"provider": "openai", "api": "chat.completions.create", "sdk": "openai", "sdk_version": "1.x",
            "runlatent": "0.3.7", "captured_at": "2026-10-02T12:00:00.000+00:00", "stream": false,
            "response_id": "chatcmpl-abc123", "finish_reason_raw": "stop",
            "usage": {"input_tokens": 120, "output_tokens": 48, "total_tokens": 168},
            "parts_dropped": 0, "items_dropped": 0}}

• request_id is the provider's response id (chatcmpl-…, msg_…, resp_…, Gemini's response_id) when it has one, else a uuid; the raw id is repeated in client.response_id.

• messages is the request as the app sent it, system prompt included, reduced to {role, content}: OpenAI messages (a tool turn is a text message; an assistant turn with tool_calls carries [tool_calls omitted]); the Responses API's instructions as system and its input string or message items (non-message items, function call outputs and the like, are skipped and counted in items_dropped; previous_response_id is recorded, the earlier turns are not fetched); Anthropic's system as system and messages (tool_result text kept, thinking blocks skipped); GenAI's config.system_instruction as system and contents (model role becomes assistant, loose strings and parts become one user turn).

• output is the whole answer text: the first choice's message.content; the Responses output's message items; Anthropic's text blocks; the first Gemini candidate's text parts (thoughts skipped). Non-text answer parts (a tool_use block, a Gemini function_call, an image) contribute nothing to it, so an answer with no text is not posted: counted skipped_tool_only when such parts were seen, skipped_empty_output otherwise (a blocked prompt). A Responses API response with status: failed or an error is not an answer and is never posted (skipped_failed_response), in the stream (response.failed) or not.

• messages is read BEFORE the SDK method runs, into fresh dicts and strings, sync and async alike. The pair is posted later (after the response, after a stream, after an await) but from that snapshot: an app that appends the assistant turn to its own list while the call is in flight or before the stream is exhausted does not leak the answer into the posted prompt. A one-shot iterable (a generator for messages, Responses input, Anthropic messages / system, GenAI contents) is not read ahead, which would empty it for the SDK: it reaches the SDK through a recording pass-through, so the provider receives exactly the elements the app produces (and, if the generator raises, the same exception at the same point; that request is counted skipped_unreadable_request), and the snapshot is what the SDK consumed.

• finish_reason is the plugin's vocabulary, so the service's finish_to_review rule reads it; the provider's word is kept in client.finish_reason_raw; a missing one stays unset so the sidecar marks finish_unknown rather than guess. The map:

  posted	OpenAI chat	OpenAI Responses (status[:reason])	Anthropic	Gemini
  stop	stop	completed	end_turn, stop_sequence	STOP
  length	length	incomplete:max_output_tokens	max_tokens, model_context_window_exceeded	MAX_TOKENS
  tool_calls	tool_calls, function_call		tool_use	MALFORMED_FUNCTION_CALL
  content_filter	content_filter	incomplete:content_filter	refusal	SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII, IMAGE_SAFETY, LANGUAGE
  abort	(stream closed early)	cancelled, in_progress, queued
  passed through		failed	pause_turn	other, unknown (OTHER, FINISH_REASON_UNSPECIFIED), blocked:<reason> for a blocked prompt

  A word not in the table passes through lower-cased, cut to the sidecar's 32 characters.

• A conversation longer than the sidecar's 256-message limit keeps its system turn and the newest turns; the number dropped is in client.messages_truncated (without the cap the sidecar answers 422 and the pair is lost).

• session_id / session_user / session_label (under runlatent.session(...)), or session_fingerprint (inferred from the conversation when no id was given), and session_turn (the count of assistant turns in messages plus one) name the call or conversation the turn belongs to ("Group the turns of a call", below).

• Multimodal and tool parts in the PROMPT (images, audio, files, inline_data, function calls) are reduced to a marker in place, [image_url omitted], so the prompt keeps its shape, and counted in client.parts_dropped and in stats()["multimodal_parts_dropped"]; in the answer they are dropped without a marker (above). The bytes never leave the process.

• usage is normalized to input / output / total tokens.

The sidecar validates the body with its own ScoreRequest; the client block is outside that contract today and is ignored by pydantic, not refused (tests/test_sdk.py posts the body to the real app). Token ids are not posted (the SDKs do not expose them).

Tag the sources: runlatent.context(docs)

Latent checks an answer against the source it was written from, so the source must be separable from the question in the posted pair: a tagged context (below, or runlatent.check(prompt, answer, context=docs)), a Knowledge: line opening the user turn, or the system message. An answer with none of these (context_source: none, a bare question or passages interleaved with no marker) is stored and shown in the console but gets no grounding verdict and is never flagged (the reply reads verdict: "unknown", unscored_reason: "no_source", the reader's number as risk_informational; the SDK result is unscored), and it is not judged, so the calibration and the first fit have fewer rows. Apps that retrieve passages and build the prompt themselves can tag them without changing the prompt:

with runlatent.context(docs):                      # a str, a list of str, or [{"title", "text"}]
    client.chat.completions.create(model="gpt-5", messages=messages)

@runlatent.context(lambda question, docs: docs)    # or a decorator: the docs from the call's own arguments
def answer(question, docs): ...

Every call captured inside posts the passages as context beside messages (at most 64 passages, 200,000 characters in all); nothing in the app's call changes; thread- and task-local, async-safe, never raises. The sources tagged when the call is made travel with it, however the stream is consumed: a stream created inside the block and iterated outside it (a FastAPI StreamingResponse, another thread or task) still posts them, and a block entered after the call does not apply to it. A redact hook sees context on the body like every other field: mask it too when the passages carry what the messages carry. runlatent.check(prompt, answer, context=...) renders its own context into the user turn as a Knowledge: line (one copy, redacted with the messages) and posts the ambient block only when no context= was given. Apps that cannot wrap the call put the passages in the system message or under a Knowledge: line (docs/hosted-score.md, "What the judge needs as the source").

Group the turns of a call: runlatent.session(...)

A voice agent or a chat app makes many LLM calls per call or conversation. Usage is per scored answer (one per turn), and the review queue can say "these answers are all from the same session" and roll a session up when the turns are tagged. Wrap the call loop once:

with runlatent.session(call_id, user_id=caller_hash):      # one block around the call loop
    for turn in call:                                      # every captured call and every check() inside is tagged
        reply = client.chat.completions.create(model="gpt-5", messages=history)
        history.append({"role": "assistant", "content": reply.choices[0].message.content})

@runlatent.session(lambda call, **kw: call.id, label="Support line")   # or a decorator: the id from the call's arguments
async def handle(call): ...

r = runlatent.check(prompt, answer, session_id=call_id, user_id=caller_hash)   # or per check, which wins over the block

Every pair posted inside carries session_id (and session_user / session_label when given) and session_turn, the count of assistant turns in messages plus one. Per-turn scoring is the unit: each turn is one scored answer, one console row and one on the bill; the session is how the rows are grouped. The queue shows a session chip on each row ("Session, turn 7 of 12"), the request page the previous and next turn in the session, GET /queue?session=<id> lists one session's turns in order (passes included), GET /sessions rolls each session up (turns, flagged turns, max risk, last verdict, open reviews) and /stats.sessions counts them (service/README.md). The optional policy rule review.session_rule.flag_once_per_session (off by default) routes a second flagged turn of a session into the session's open review instead of opening a second queue row.

The ids are your own opaque strings, stored as given and shown in the console: they must not be personal data. Hash a phone number or an account id before passing it as user_id (caller_hash above); the same goes for session_id and label. A hash-only plan gains nothing readable from them beyond these ids. One line each, up to 256 characters (cut, never refused); thread- and task-local (contextvars), async-safe, never raises into the app.

Inferred sessions. With no explicit id, an app that passes the conversation history each turn (messages) gets session_fingerprint: the sha256 of the system turn plus the first user turn, each whitespace-normalized the way the pair fingerprint is, computed on the list the app sent (before the 256-message cap trims old turns), so it stays the same for the whole conversation. An explicit id wins over it. Two conversations that open with the same system prompt and the same first user message share a fingerprint, so pass an explicit id when that matters (a FAQ bot whose first turn is "Hi"); the service never merges reviews on a fingerprint, only on an explicit id. Inference is on by default and is the one thing the posted body gains on upgrade; turn it off with runlatent.configure(infer_session=False) or LATENT_INFER_SESSION=0 (then a conversation without an explicit id posts session_turn only). A bare check(prompt, answer) has no conversation to infer from: only an explicit or ambient id names its session. The block is contextvars-scoped: asyncio.to_thread (and acheck) inherit it, a ThreadPoolExecutor worker does not (pass session_id= to a check() run on a pool). The streaming read records its pair through the reader's stream session, whose open carries the same session fields (SDK 0.3.5: the id, user and label of the block the call was made in, else the inferred fingerprint, plus the turn), so a streamed turn is grouped like a posted one. Under the policy rule, the merge is by session id across models: one call served by two models is one review.

What is never posted

• API keys, headers, the request's other parameters (temperature, tools, response formats, metadata): only model and the message text are read.
• Image, audio and file bytes or URLs (reduced to a marker, see above).
• Thinking / reasoning blocks (Anthropic thinking, Gemini thought parts, Responses reasoning items).
• Anything when the call failed: the SDK's exception propagates unchanged and no pair exists.
• Anything the redact hook returned None for, or raised on (a failed masker never lets the unmasked pair through; the drop is counted).

Streaming

A stream is returned to the app through a thin proxy that yields the same items and delegates every other attribute (.response, .close(), with …), and the pair is posted once, at the end: when the iterator is exhausted (OpenAI chat chunks and Responses events, Anthropic raw events, Gemini chunks are assembled into the full text; the last chunk's finish and usage are taken), or when the proxy's own close() or context exit runs before that, in which case the partial text is posted with finish_reason: "abort" and client.stream_complete: false. A stream the app simply stops reading (for chunk in s: break with no close() and no with on s) is never posted: it is counted stream_unfinished when the proxy is garbage-collected (a finalizer counts, it does not post). OpenAI's own helper managers, client.chat.completions.stream() and client.responses.stream(), go through the patched create and capture once when run to exhaustion; their __exit__ closes the HTTP response, not the stream, so an early exit from one of them is the unfinished case above, not abort. One consequence: chat.completions.stream() can raise LengthFinishReasonError from its own parsing after the final chunk arrived; the raw stream under it was exhausted, so the pair IS posted, as length, although the app got an exception rather than a completion (E-R149). abort is only ever the app closing a healthy stream early: a stream that raised mid-way, or a with block an exception left (the SDK raising on an error event, Anthropic's overloaded_error included, or the app raising inside the block), posts nothing and is counted stream_errored. Anthropic's messages.stream() manager hands the app the SDK's MessageStream through a thin proxy (same events, text_stream, get_final_message() and every attribute delegated) that remembers a failure raised while reading, so an overloaded_error the app catches inside the block still posts nothing; a clean exit reads current_message_snapshot (no extra network read), and a healthy stream the app stops reading early posts the partial text as abort. Only the first choice / candidate is assembled when n > 1.

Streaming read: the risk while the answer streams (follows the console's policy)

runlatent.auto_instrument()                      # the streaming read follows the project's policy (SDK 0.3.5):
                                                 # on while the console says Hold, or Scoring: sentence by sentence
runlatent.auto_instrument(stream_read=True)      # or wrap_openai(client, stream_read=True): PIN it on (or False: off),
                                                 # process-wide, whatever the policy says; the default None leaves a pin as it is

stream = client.chat.completions.create(model="gpt-4.1", messages=msgs, stream=True)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
    read = runlatent.current_read()              # the live read of THIS stream, or None
    if read and read["flagged"]:
        print("\n[withheld: this answer is being flagged]")
        stream.close()                           # the app decides; runlatent never withholds a chunk itself
        break

No code flag to remember (SDK 0.3.5). The read is on exactly while the polled policy document says stream_read.enabled: true (runlatent._enforce's poller, the same document the enforcement reads; nothing is waited for): the console's Policy page writes it with every Hold ("When Latent flags an answer: Hold it and say / Hold it and ask a stronger model") and with Scoring: sentence by sentence while streaming under Advanced, and clears it with Deliver it and log it / at the end of each answer. The next stream after a poll (60 s, or at once on the first document) follows the new setting. stream_read=True / False is the app's pin and wins over the policy either way (stats()["stream_read_source"] reads "override", else "policy" when the document turned it on, else "off"); set_stream_read(None) drops the pin. With auto_instrument(enforce=False) (or LATENT_SDK_ENFORCE=off) there is no poller and so no policy-driven read: pin it if you want it there. Against a destination without a policy route (a sidecar: 404) the read stays off unless pinned.

With the read on, a streamed call is also fed, delta by delta, to Latent's reader as the app reads it: the reader runs its streaming read session (<LATENT_API_URL>/stream/open, /feed, /close: a sidecar with LATENT_STREAM_READ=1; latent_sidecar/README.md, "The streaming read session"), which scores every token of the answer so far and runs the plugin's early-stop rule on the running mean. runlatent.current_read() returns that read for the stream the calling code is iterating (the proxy's latent_read() is the same object): risk (the artifact's quantity at the last read), mean_risk (the running mean, what the rule reads), n_tokens, threshold, rule, early_stop (the reader's decision record once the rule fired, else None), flagged (the decision fired, or the running mean is at or above the cut), status (pending before the first delta reached the reader, reading, unavailable, closed). The read lags the stream by the reader's step and one round trip (a few tokens); the feeding runs on a daemon thread behind a queue, so the app's iteration never waits on it, and deltas that arrive while a feed is in flight ride the next one. The SDK never withholds: every chunk the provider yielded reaches the app when it was yielded; stopping is the app's break (posted as finish_reason: "abort", like any early close).

The session's open carries the turn's session fields (runlatent.session(...)'s id, user and label, captured where the call was made, or the inferred fingerprint, plus session_turn; the same keys the one-pass post sends, "Group the turns of a call"), so a streamed turn is grouped with the session's other turns in the console. The session's close is the record: the reader writes the event with the per-token trace (meta.capture: "sidecar-stream", meta.trace, meta.early_stop), so nothing is posted to /score for that stream. A session that cannot be opened (the reader down, at its cap, the stream read off: 404), fails on the way or is refused at close falls back to the one-pass /score post of the whole stream, counted stream_read_fallback; nothing is lost. Counters in stats(): stream_read (the effective switch, on / off) and stream_read_source (override / policy / off), stream_read_sessions, stream_read_fed, stream_read_stops, stream_read_closed, stream_read_unavailable, stream_read_failed, stream_read_fallback, stream_read_aborted (a stream that raised mid-way, whose capture failed, or that the app DROPPED without close() / with (a bare break, an exception in a plain for body, a dropped async for) releases its session unscored when the proxy is collected: nothing is recorded, as the one-pass path records nothing for such a stream, and current_read() is None for it from then on), stream_gateway_read (below). current_read() is task-scoped under asyncio: a task that iterates no stream reads None, never another task's stream; and it is None once the stream has ended, whichever way it ended.

With runlatent.check(): a stream read and a check() of the same pair are ONE reader pass in either order: check() by the provider's response id or by the conversation adopts the stream's close reply (check_adopted), and a stream whose pair check() already recorded releases its session unscored (skipped_checked). The reader itself answers a /score of a request id its stream read just closed from memory (deduplicated: "stream_close", bound to the same messages and output; other content under that id is read fresh and marked conflicting_replay), so the fallback after a close reply that timed out on this side costs no second pass either.

Behind Latent's gateway with the shadow read on (LATENT_GATEWAY_STREAM_READ=shadow): the gateway's reader already records the stream and its trailing chunk's latent block says so. Equal ids inside one provider stream do not authenticate a gateway, so the SDK adopts that tail as the record only when the app has configured a TRUSTED gateway: runlatent.configure(gateway_hosts=["gateway.internal"]) or LATENT_GATEWAY_HOSTS=gateway.internal,... (comma list), the provider client's base URL host (base_url on the OpenAI / Anthropic client; http_options.base_url on a google.genai.Client, default generativelanguage.googleapis.com; read at every call, so a client whose base_url changes after the wrap is judged by where it points now) is one of them, AND the tail carries a completed scored read (stream_read.status: "read" with a risk) bound to this request (its request_id equals the stream's own id; the gateway stamps it on every shadow tail). Then the SDK releases its own session unscored and posts nothing (stream_gateway_read), whatever stream_read is. With no trusted gateway configured (the default) every tail is ignored and the SDK records as usual, counted stream_gateway_tail_untrusted (the service keeps the first row per request id, so a gateway + SDK pair is still one stored record); a tail from a trusted host that is not this request's is stream_gateway_tail_unbound. A chunk carrying stop_reason: "latent_early_stop" is read as the plugin's early stop (finish_reason: "early_stop", its replacement text not the model's answer), not as a length finish. Behind a gateway running the shadow read, leave stream_read off and read the tail's latent block instead; stream_read=True is for the app that calls the provider directly.

Redaction: redaction and the live stream read are mutually exclusive: a redacting hook (runlatent.configure(redact=...)) disables live reads, and the recorded pair is redacted. A whole-pair hook has no safe incremental contract (a lookahead decides on text that arrives later), and text once fed to the reader cannot be retracted, so with redact= configured the stream's deltas are never fed; the fully redacted pair is posted once at the end exactly as without stream_read, counted stream_read_redact_fallback.

Not fed: Anthropic's messages.stream() manager (with client.messages.stream(...) as s) reads current_message_snapshot at exit and has no delta path through the proxy, so it is scored once at the end as before and current_read() is None inside it; the raw event stream (messages.create(..., stream=True)) is fed. The Responses API and Gemini streams are fed from the same assembled text pieces as the OpenAI chat stream. On the hosted service (docs/hosted-score.md, "The streaming read") the /stream/* routes are carried for Pro and Team with usage billing on, for a project that turned the streaming read on in the console (Policy > "When Latent flags an answer": either Hold, or Advanced > Scoring: sentence by sentence while streaming; the same stream_read.enabled the SDK follows); a Free key, usage billing off, or a project with the read off falls back to the one-pass post on every stream (counted stream_read_fallback). Every sentence read there is one scored answer on the bill beside the answer, whether or not the stop below is on.

Against the hosted route, with usage billing on, the read can stop the stream. With the project's policy.stream_stop on (the console's Policy page: either Hold choice sets it; the cut under Advanced > "Pin the cut"), a feed reply may carry stop when the sentence head's read over the completed sentences crosses the project's cut; the proxy then ends the model's stream, delivers the policy's fallback text as a final chunk where the provider's chunk shape can carry it (OpenAI chat chunks: action: stop) or ends the iteration with nothing more (action: hold), and closes the session with finish_reason: "early_stop"; runlatent.last_decision() and the read's action say what happened, counted stream_stop_stop / stream_stop_hold; auto_instrument(enforce=False) leaves the stream untouched (stream_stop_ignored). This is the one case where the SDK ends a stream itself, and the customer turns it on.

Redaction (default: none)

By default the SDK posts the text as the app sent and received it, because masking changes what Latent reads: E-R143 (2026-10-01) ran a default de-identification tool over logged prompts and answers masked separately and a third of the good answers turned into judged failures (the answer named "Felica Cannon", the masked source only "Felica"), the probe lost up to 0.07 AUROC and the flagged list overlapped the unmasked one about 50%. Identifiers-only masking with ONE dictionary derived from the source and applied to prompt and answer alike kept the judge at 0.96 and the read within 0.02. If you must mask before posting, do that, and do it in one place:

def redact(body):
    table = my_entities(body["messages"])             # names from the source, one dictionary
    body["messages"] = [dict(m, content=mask(m["content"], table)) for m in body["messages"]]
    body["output"] = mask(body["output"], table)      # the same table on the answer
    return body                                        # or None to skip this pair

runlatent.auto_instrument(redact=redact)

The hook sees the full body (the client block included) and may rewrite or drop it. Prefer raw text under the processing terms (retention is the service's policy.retention) and mask after the read where a report needs it.

Counters and shutdown

runlatent.stats()
# {"enabled": true, "api_url": "...", "patched": ["openai:Completions.create", ...],
#  "captured": 120, "posted": 118, "failed": 0, "dropped": 2, "queued": 0,
#  "skipped_empty_output": 3, "multimodal_parts_dropped": 7, "redact_skipped": 0,
#  "checked": 40, "check_scored": 38, "check_adopted": 30, "check_unscored": 1, "check_refused": 1, "skipped_checked": 2,
#  "errors": {"post:ConnectionRefusedError": 0}, "poster": {... the poster's own counters ...}}
runlatent.flush(timeout=5.0)   # at shutdown: wait for the queue to be attempted

enforce_read counts the synchronous reads the policy asked for ("Act from the console"), enforce_<action> the decisions (enforce_abstain, enforce_regenerate, enforce_route, enforce_judge, enforce_none, enforce_hold, enforce_regenerate_failed, enforce_route_failed), enforce_fail_open / enforce_fail_closed the reads that did not come back in the budget by how they ended, enforce_replace_failed a response shape the SDK could not rewrite (delivered as it was, said so in latent.reason), enforce_reported / enforce_report_failed / enforce_report_dropped the decision reports; stats()["enforce"] carries enabled, the poller's reading (policy: url, version, fetched, not_modified, failures, unauthorized, last_status) and last_decision. checked counts check() calls; check_scored those answered with a risk (check_adopted of them read from the instrumentation's own POST rather than a second one), check_unscored the empty answers, check_refused the refusals and transport failures (by class under errors, check_post:TimeoutError), skipped_checked the captured pairs the poster did not post because a check recorded them, check_detail_reposted the check(detail=True) calls that posted their own read because the captured one had already gone out without detail.

captured counts pairs assembled; posted / failed / dropped / queued are the poster's (latent_vllm.events.SinkPoster through latent_sidecar.tap.FireAndForget: daemon thread, bounded queue, three attempts per pair, no spill file: a pair that cannot be delivered is lost and counted). dropped includes pairs captured with no destination configured. errors is keyed by stage and exception class (observe:… a shape the adapter could not read, redact:…, post:…, stream_feed:…); nothing in it ever reached the app. Only the class is logged (at DEBUG), never the exception's message: a masker that raises with its input in the message must not put answer text in your log.

Limits

• The streaming read is process-wide (one reader client for the API base; the policy's switch or the app's pin), reads sample 0 of a stream only, and needs a reader with the streaming read session; the Responses, Anthropic and Gemini streams are fed from the same assembled text pieces as the OpenAI chat stream.
• Only the methods above are patched. chat.completions.parse() and responses.parse() post through their own path and are not captured; beta.*, the Assistants and Realtime APIs are not in this version; wrap a gateway or use latent_sidecar.tap for those. A client the SDK does not patch (Replicate, SageMaker invoke_endpoint, Bedrock invoke_model, in-process transformers) goes through runlatent.check(prompt, answer, context=...): docs/sdk-providers.md has the matrix. Captured because they route through the patched methods: chat.completions.stream() and responses.stream() (exhaust-to-capture, see Streaming) and Gemini chats.send_message.
• Arguments must be passed by keyword (the SDKs require it; a positional messages is not read), with two exceptions: InferenceClient.chat_completion(messages, ...) takes messages positionally and ollama's chat(model, messages) / generate(model, prompt, suffix) take theirs so, and the SDK binds them (class patch and wrap_huggingface / wrap_ollama alike; docs/sdk-providers.md).
• with_raw_response.create(...) and with_streaming_response.create(...) return the HTTP response object, not the parsed completion; reading it would consume a streaming body, so those calls are not captured and are counted under stats()["skipped_raw_response"].
• At a normal interpreter exit an atexit handler waits up to LATENT_SDK_EXIT_FLUSH_S seconds (default 2.0) for the queued pairs to be posted, so a short script that never calls flush() still delivers; with it set to 0 nothing waits and pairs still queued at exit are lost without a counter (the process is gone). A kill -9 or an os._exit() loses them either way: call runlatent.flush() in your shutdown path when it matters.
• The proxy around a stream is not an instance of the SDK's Stream class; isinstance checks on it fail (attribute access does not).
• There is no on-disk spill: a Latent outage longer than the retries loses those pairs, counted.
• max_queue counts pairs, not bytes, behind one serial poster thread (ingest tops out near 1 / POST latency pairs per second). A full queue of 10,000 pairs with 100 KB RAG prompts is about 1 GB held in the process before anything is dropped: size max_queue to the burst you are willing to hold in memory, and lower it on memory-tight hosts.
• Multi-choice (n > 1) and multi-candidate responses post the first only.
• Gemini thinking models count their thought tokens against max_output_tokens: a short budget yields many empty answers (not posted, skipped_empty_output) and truncated ones, posted as length and routed to review by the service's default finish_to_review rule (E-R149's first Gemini run at 160 tokens produced 44 of them). Give thinking models a budget that covers the thinking, or disable it, before reading the review queue as a quality signal.