auscope

Fire-and-forget LLM output auditing. Wrap the LLM client you already use — no call-site rewrites — and every response gets sent to an Auscope audit server in the background. Get back a verified / uncertain / unreliable verdict, a confidence score, reasoning, and real per-call cost, without blocking your response to the user.

import anthropic
from auscope.adapters.anthropic import AuscopeAnthropic

client = AuscopeAnthropic(
    anthropic.AsyncAnthropic(api_key="..."),
    audit_url="https://api.your-auscope-instance.com",
    api_key="asc_live_...",
)

response = await client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
# response is exactly what anthropic.AsyncAnthropic would have returned —
# the audit fired in the background, doesn't block this call.

verdict = await client.last_verdict()
print(verdict.verdict, verdict.confidence_score, verdict.cost_usd)

Install

pip install auscope-sdk[anthropic]      # or [openai], [azure], [google], [langchain], [maf], [all]

Each extra installs the one provider SDK you need. Core install (pip install auscope-sdk) only pulls httpx + pydantic — bring your own provider SDK if you don't want an extra. The package is auscope-sdk on PyPI; you still import auscope in code.

Supported adapters

Every adapter is a thin subclass — anything you'd normally call on the real SDK client still works via passthrough (__getattr__). Streaming (stream=True) is supported on the Anthropic and OpenAI/OpenRouter adapters: chunks pass through unchanged, the audit fires once with the full accumulated text after the stream is exhausted.

Sync clients

If your codebase uses the synchronous SDK variants (openai.OpenAI, anthropic.Anthropic) instead of the Async* ones:

from openai import OpenAI
from auscope.sync import AuscopeOpenAISync

client = AuscopeOpenAISync(OpenAI(api_key="..."), audit_url="...", api_key="asc_live_...")
response = client.chat.completions.create(model="gpt-4o", messages=[...])
verdict = client.last_verdict()  # blocks until the audit completes

Audits still fire in the background on a shared thread — the call returns immediately. No streaming support in the sync variant yet.

Constructor options

Every adapter (and auscope_watch) accepts:

Per-call overrides: pass auscope_system_prompt=... / auscope_chat_history=... as extra kwargs to any wrapped call (create_message, create_completion, etc.) to override the instance defaults for that one call — useful for a single client instance serving a multi-tenant bot with different system prompts per request. These are popped before forwarding to the real SDK, so they never reach the provider.

Reading the verdict

await client.last_verdict() (or client.last_verdict() for sync clients) returns the oldest not-yet-consumed audit, FIFO per client instance:

verdict = await client.last_verdict()
verdict.verdict            # "verified" | "uncertain" | "unreliable"
verdict.confidence_score   # 0.0-1.0
verdict.reasoning          # chairman's explanation
verdict.cost_usd           # real $ cost of the audit council + chairman calls
verdict.model_calls        # list[ModelResponse] — per-auditor tokens/cost/latency

If you don't need the verdict inline, don't call last_verdict() — the audit still runs and fires on_audit_error on failure; nothing blocks.

No SDK for your provider?

Skip the adapter, audit directly:

from auscope.base import AuscopeBase

client = AuscopeBase(audit_url="...", api_key="asc_live_...")
verdict = await client.audit_direct(query="What is the capital of France?", llm_response="Paris.")

Development (this monorepo)

This package lives at sdk/ in the Auscope monorepo and is a uv workspace member — the root project depends on it via auscope = { workspace = true }, so import auscope works locally without publishing. Tests: sdk/tests/ (uv run pytest sdk/tests). Live provider tests are skipped unless the matching *_API_KEY env var is set (see sdk/.env.example).

Publishing

cd sdk
uv build
uv publish   # needs PYPI_TOKEN / --token