tellproof (Python SDK)

One line to send evidence telemetry from an AI product to the local Tellproof evidence agent.

pip install tellproof

import tellproof
tellproof.init()

The SDK records one span per model call with provider, model, region, token counts, operation, timing and error status, and sends it as OTLP JSON to the agent on http://127.0.0.1:4318 (or TELLPROOF_AGENT_URL). The agent aggregates per hour and signs the aggregates: it keeps them in a local evidence log (tellproof-agent local) or sends them to Tellproof (tellproof-agent connect). The SDK never reads prompts, completions, function arguments, return values or exception messages, so there is no content to filter.

• No dependencies: standard library only, Python 3.10+.
• Never slows or breaks the app: spans go to a bounded background buffer; when it is full, spans are dropped and counted (tellproof.stats()). SDK errors are swallowed and counted.
• TELLPROOF_DISABLED=1 turns it off. init() is safe to call more than once.
• Proxies are never used: the agent is local.
• Works across threads, asyncio tasks and forked workers (gunicorn, uWSGI).

Automatic instrumentation

init() instruments the client libraries that are installed. Nothing to configure:

Per call it reads the request's model and stream flag, and from the response only the model, the usage block, the stop reason and Bedrock's token-count headers. Input tokens follow what the provider bills (Anthropic: uncached + cache writes + cache reads; Bedrock: inputTokens, as CloudWatch counts them), so Tellproof can reconcile traced against billed tokens.

Things to know:

• Streams are recorded when they end, are closed, or leave a with block, once, with the usage they reported. A stream dropped half-read is recorded on the next call or flush.
• OpenAI streamed chat completions report usage only if asked. tellproof.init(stream_usage=True) sets stream_options={"include_usage": True} when the app has not set it; the stream then ends with one extra chunk with no choices. Off by default, so the app's streams are unchanged.
• Bedrock guardrails: a Converse call with guardrailConfig is a guarded call, block when the stop reason is guardrail_intervened, else allow.
• invoke_model_with_response_stream reports its token counts only inside the streamed body, which the SDK never reads, so those calls are counted without tokens. with_raw_response calls are counted without tokens too.
• tellproof.init(instrument=False) turns automatic instrumentation off; the helpers still work.

What the helpers record

Tellproof's evidence objectives need a few facts that no LLM client knows. Each helper is a context manager (with or async with) and a decorator.

with tellproof.trace(), tellproof.prompt("claims-triage", "14"), tellproof.personal_data(redacted=True):
    with tellproof.guardrail() as g:
        if not input_policy_ok(request):
            g.block()          # recorded as one blocked call; the model was never called
            return refusal()
        with tellproof.llm_call("openai", "gpt-4.1-mini", region="us-east-1") as c:
            r = client.chat.completions.create(...)
            c.usage(r.usage.prompt_tokens, r.usage.completion_tokens)
        g.flag() if needs_review(r) else g.allow()

@tellproof.tool("issue_refund", high_impact=True)
def issue_refund(claim): ...

Names and versions are capped at 128 characters; anything that is not a string or an integer is ignored.

Calls inside a guardrail scope are held until the scope ends, so a decision made after the call (an output check) still applies. A block with no call inside counts as one blocked call; pass provider= and model= to attribute it.

Guardrail canaries

Show that your guardrail catches something, not only that it ran. The SDK passes synthetic, known-bad probes (prompt injection, jailbreak, made-up personal data, a documented example AWS key) to your own guardrail check and records only the kind and whether it was caught (EO-CAN-01):

tellproof.schedule_canaries(lambda text: my_guardrail.scan(text).blocked, every=3600)  # now, then hourly
tellproof.run_canaries(check)          # once: {"prompt_injection": True, ...}; arun_canaries for an async check
tellproof.canary("toxicity", True)     # a canary you run with your own probe

The check returns True, "block" or "flag" when it would stop the text; anything else, or an exception, counts as missed. Pass probes to the guardrail only, never to the model. The probes never leave the process (tellproof.PROBES); with the SDK off the check is never called.

Already on OpenTelemetry?

You can skip the SDK: point an OTLP/HTTP JSON exporter at the agent (http://127.0.0.1:4318/v1/traces) and set the tellproof.* attributes yourself (spec/semconv-map.json). Do not run both for the same calls, or they are counted twice.

Development

python3 -m unittest discover -s sdk/python/tests
python3 sdk/python/tests/scenario.py   # rewrite sdk/testdata/python-export.jsonl