traceleakcheck 0.1.0

Quick start · How it works · Safety filter · Real-world test · Survey

Your AI app's monitoring data (OpenTelemetry traces and logs) records prompts, tool calls and errors, and sends them to a dashboard. Secrets ride along: an API key in an error message, a password a user pasted into the chat, a token in a URL. Redaction code passes its unit tests while the key still leaks through a field nobody thought about.

TraceLeakCheck plants a fake secret, captures everything your app really exports, and tells you in plain English exactly where it leaked.

What you get

Quick start

pip install traceleakcheck

def test_agent_does_not_leak_key(canary):
    key = canary("openai").value          # a fresh fake secret; this test is now checked
    run_agent("summarize this", api_key=key)

pytest --traceleak                                 # plain-English results in the terminal
pytest --traceleak --traceleak-html=report.html    # + a report page for your browser
pytest --traceleak -v                              # + exact technical field paths

If your app configures OpenTelemetry from the standard OTEL_EXPORTER_OTLP_* settings, it needs no changes. Traceloop's TRACELOOP_BASE_URL is set too. If your app builds a provider by hand, register it with traceleak.watch(provider).

How it works

1. Before your conftest.py loads, --traceleak starts a receiver and points the app's exporters at it.
2. Each test that asks for a canary gets unique fake secrets in realistic formats: OpenAI, Anthropic, AWS, GitHub, JWT or password.
3. After the test and its fixtures finish, a control marker is sent through every configured pipeline (traces, logs, metrics), everything is flushed, and every exported field is searched.

Each test gets one verdict:

pytest shows a failed check as ERROR at teardown, because it runs after the fixtures finish. It works with pytest -n (xdist). In GitHub Actions the verdicts also go to the job summary, and there's a ready-made action.

Fix it: the safety filter

from traceleakcheck.guard import scrub

tracer_provider.add_span_processor(BatchSpanProcessor(scrub(OTLPSpanExporter())))
logger_provider.add_log_record_processor(BatchLogRecordProcessor(scrub(OTLPLogExporter())))

Before anything is exported, the filter removes:

• Known secrets: the values of environment variables named like *KEY*, *TOKEN*, *SECRET* or *PASSWORD*, plus anything registered with add_secret(). Their base64, URL and hex forms are removed too.
• Well-known key formats: OpenAI, Anthropic, AWS, GitHub, Google, Slack, JWT and Bearer tokens, and credential query parameters such as ?key=.

It covers span names, attributes, exception events, status messages, log bodies and log attributes.

A filter can't recognize a secret that matches none of these rules, such as an unusual password a user types. That's why the two go together: the filter is the fix, and pytest --traceleak is the proof.

Tested on real code

We pointed TraceLeakCheck at the OpenTelemetry Demo's AI shop assistant (LangGraph + Traceloop), unmodified, talking to a fake LLM:

One secret becomes many copies, because every layer of an agent (chat step, agent step, workflow) records its own copy. The run also surfaced a real integration bug in TraceLeakCheck: the Traceloop SDK ignores OTEL_EXPORTER_OTLP_*. That's now fixed.

Survey: do GenAI instrumentations leak secrets?

survey/FINDINGS.md runs the OpenAI SDK under OTel openai-v2, OpenLLMetry, OpenInference, Logfire and httpx instrumentation, across 6 leak scenarios:

• No content opt-out we tested stops a secret in an error message. That includes TRACELOOP_TRACE_CONTENT=false, OPENINFERENCE_HIDE_* and NO_CONTENT.
• ?key= query auth is exported verbatim in http.url.
• Logfire's default scrubbing skips LLM messages and exception messages, even when they contain "API key".
• OTel openai-v2's documented legacy CAPTURE_MESSAGE_CONTENT=true falls back to NO_CONTENT on util-genai 1.x, with only a log warning.
• Streamed replies are reassembled, so a secret split across chunks still leaks. But OTel openai-v2 never ends the span of an abandoned stream, so its contents are never exported. traceleakcheck now reports that as incomplete instead of a false clean.

Limits

• Python and pytest only for now.
• Sampling is forced on with OTEL_TRACES_SAMPLER=always_on. An app that passes an explicit sampler in code can still drop the span that leaks.
• Telemetry that isn't OTLP, such as metrics read by a Prometheus reader, can't be checked. Skip it with --traceleak-ignore=metrics.

Compared to Weir

Weir uses the same canary idea, and it covers more ground: it runs agent flows against a fake model and checks console output and network requests as well as telemetry. TraceLeakCheck focuses on the OTLP export boundary, needs no fake model, and ships a fix (scrub()) alongside the check.

Weir's README says it "finds whole values only, not hashed, split or truncated ones". TraceLeakCheck catches truncated secrets. It also has tests for base64 at any byte offset, secrets used as attribute keys, protobuf fields the decoder drops, and mixed-case hex. As of 2026-11-01, Weir's README doesn't say either way whether it handles these.

from traceleakcheck import OTLPReceiver, make_canary, scan

with OTLPReceiver() as rx:
    canary = make_canary("openai")
    # point your OTLP exporter at rx.endpoint, run your app with canary.value as the API key...
    findings = scan(rx.snapshot(), [canary])

Develop

uv sync
uv run pytest