auscope-sdk 0.1.0

Fire-and-forget LLM output auditing — wrap your existing SDK client, get a verified/uncertain/unreliable verdict on every response.

auscope

Fire-and-forget LLM output auditing. Wrap the LLM client you already use — no call-site rewrites — and every response gets sent to an Auscope audit server in the background. Get back a verified / uncertain / unreliable verdict, a confidence score, reasoning, and real per-call cost, without blocking your response to the user.

import anthropic
from auscope.adapters.anthropic import AuscopeAnthropic

client = AuscopeAnthropic(
    anthropic.AsyncAnthropic(api_key="..."),
    audit_url="https://api.your-auscope-instance.com",
    api_key="asc_live_...",
)

response = await client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
# response is exactly what anthropic.AsyncAnthropic would have returned —
# the audit fired in the background, doesn't block this call.

verdict = await client.last_verdict()
print(verdict.verdict, verdict.confidence_score, verdict.cost_usd)

Install

pip install auscope-sdk[anthropic]      # or [openai], [azure], [google], [langchain], [maf], [all]

Each extra installs the one provider SDK you need. Core install (pip install auscope-sdk) only pulls httpx + pydantic — bring your own provider SDK if you don't want an extra. The package is auscope-sdk on PyPI; you still import auscope in code.

Supported adapters

Provider	Import	Wraps	Drop-in method
Anthropic	auscope.adapters.anthropic.AuscopeAnthropic	anthropic.AsyncAnthropic	client.messages.create(...)
OpenAI	auscope.adapters.openai.AuscopeOpenAI	openai.AsyncOpenAI	client.chat.completions.create(...)
OpenRouter	auscope.adapters.openrouter.AuscopeOpenRouter	openai.AsyncOpenAI (OpenRouter base_url)	client.chat.completions.create(...)
Azure AI Inference	auscope.adapters.azure.AuscopeAzure	azure.ai.inference.aio.ChatCompletionsClient	client.complete(...)
Google Generative AI	auscope.adapters.google.AuscopeGoogle	google.generativeai.GenerativeModel	client.generate_content_async(...)
LangChain	auscope.adapters.langchain.AuscopeLangChain	any LangChain BaseChatModel	client.ainvoke(...)
MAF (Multi-Agent Framework)	auscope.adapters.maf.AuscopeMAF	an agent_framework.Agent	client.run(...)
Any provider	auscope.adapters.raw.auscope_watch	a plain async def fn(prompt) -> str	@auscope_watch(...) decorator

Every adapter is a thin subclass — anything you'd normally call on the real SDK client still works via passthrough (__getattr__). Streaming (stream=True) is supported on the Anthropic and OpenAI/OpenRouter adapters: chunks pass through unchanged, the audit fires once with the full accumulated text after the stream is exhausted.

Sync clients

If your codebase uses the synchronous SDK variants (openai.OpenAI, anthropic.Anthropic) instead of the Async* ones:

from openai import OpenAI
from auscope.sync import AuscopeOpenAISync

client = AuscopeOpenAISync(OpenAI(api_key="..."), audit_url="...", api_key="asc_live_...")
response = client.chat.completions.create(model="gpt-4o", messages=[...])
verdict = client.last_verdict()  # blocks until the audit completes

Audits still fire in the background on a shared thread — the call returns immediately. No streaming support in the sync variant yet.

Constructor options

Every adapter (and auscope_watch) accepts:

Kwarg	Default	Meaning
audit_url	http://localhost:8000	Your Auscope server.
api_key	None	Auscope API key (asc_live_...), sent as Authorization: Bearer.
domain	"general"	Audit domain hint (e.g. "medical", "legal") passed to the council.
allow_search	False	Let auditor models use web search to verify claims.
models	server default	Override which council models audit this client's calls.
chairman_model	server default	Override the chairman model.
system_prompt	None	Your bot's system prompt, included as audit context.
chat_history	None	Prior conversation turns, included as audit context.
on_audit_error	None	Callable[[Exception], None] — called if a background audit fails, even if you never call last_verdict().
sample_rate	1.0	Fraction of calls to audit (0.0–1.0). Below 1.0, skips auditing at random — useful for high-volume endpoints where auditing every call is too expensive.
max_retries	2	Retries on transient network errors / 5xx from the audit server.
retry_backoff_base	0.5	Seconds, doubles each retry.

Per-call overrides: pass auscope_system_prompt=... / auscope_chat_history=... as extra kwargs to any wrapped call (create_message, create_completion, etc.) to override the instance defaults for that one call — useful for a single client instance serving a multi-tenant bot with different system prompts per request. These are popped before forwarding to the real SDK, so they never reach the provider.

Reading the verdict

await client.last_verdict() (or client.last_verdict() for sync clients) returns the oldest not-yet-consumed audit, FIFO per client instance:

verdict = await client.last_verdict()
verdict.verdict            # "verified" | "uncertain" | "unreliable"
verdict.confidence_score   # 0.0-1.0
verdict.reasoning          # chairman's explanation
verdict.cost_usd           # real $ cost of the audit council + chairman calls
verdict.model_calls        # list[ModelResponse] — per-auditor tokens/cost/latency

If you don't need the verdict inline, don't call last_verdict() — the audit still runs and fires on_audit_error on failure; nothing blocks.

No SDK for your provider?

Skip the adapter, audit directly:

from auscope.base import AuscopeBase

client = AuscopeBase(audit_url="...", api_key="asc_live_...")
verdict = await client.audit_direct(query="What is the capital of France?", llm_response="Paris.")

Development (this monorepo)

This package lives at sdk/ in the Auscope monorepo and is a uv workspace member — the root project depends on it via auscope = { workspace = true }, so import auscope works locally without publishing. Tests: sdk/tests/ (uv run pytest sdk/tests). Live provider tests are skipped unless the matching *_API_KEY env var is set (see sdk/.env.example).

Publishing

cd sdk
uv build
uv publish   # needs PYPI_TOKEN / --token