tracedrift

Lightweight tracing and divergence diagnosis for LLM agent chains.

When an agent chain does something wrong, the failure usually surfaces several steps away from its cause — a retrieval step silently returns nothing, an LLM call improvises around it, and by the time you see the bad final answer you're staring at raw JSON logs trying to reconstruct what happened. tracedrift gives you a decorator-based tracer, a queryable SQLite trace store, a diagnosis pass that flags where a chain likely diverged from its normal behavior (not just where it crashed), and a way to replay a trace with one step's output swapped out to see if the rest of the chain recovers.

No mandatory cloud dependency, no framework lock-in — it works with raw API calls, LangChain, DSPy, or anything else that's just Python functions.

See docs/USAGE_GUIDE.md for instrumentation patterns, how to get useful signal out of diagnose, async/replay caveats, and performance notes — this README is the quickstart, that's the deep dive.

Install

pip install -e .

(PyPI release TBD — this is a fresh package.)

Quickstart

import tracedrift

def retrieve(query):
    with tracedrift.span("retrieve", input={"query": query}) as s:
        s.output = my_vector_db.search(query)
    return s.output  # read back so replay overrides work — see "Replay" below

def call_llm(docs):
    with tracedrift.span("llm_call", input={"docs": docs}) as s:
        s.output = my_llm.complete(docs)
    return s.output

@tracedrift.trace_agent(name="qa_agent")
def run_agent(query):
    docs = retrieve(query)
    return call_llm(docs)

run_agent("what's our refund policy?")

Traces are written to tracedrift.db (SQLite) by default. Change it with tracedrift.configure("path/to/db") before running.

Inspecting traces

tracedrift list                    # recent traces
tracedrift show <trace_id>         # span tree with timing and status
tracedrift diagnose <trace_id>     # where did this trace likely go wrong?

diagnose doesn't just look for exceptions. It compares each span's output against that span's own history (by name) across prior traces and flags:

1. Explicit errors — a span that raised. Highest confidence, reported first.
2. Silent empty output — a span returned None/[]/"" when it historically almost never does. This catches the common case where nothing crashes but a step quietly failed.
3. Type drift — a span's output type differs from what it's always returned before (e.g. usually a list, this time a str).

These are intentionally simple, interpretable heuristics rather than an ML classifier — the goal is that you can trust why something got flagged, not just that it was.

Replay with intervention

Once you suspect a step, you can rerun the same call with that step's output forced to something else, to check whether the rest of the chain would have recovered:

result = tracedrift.replay_with_intervention(
    run_agent,
    args=("what's our refund policy?",),
    overrides={"retrieve": [<docs you think it should have found>]},
)

Important caveat: this overrides what gets recorded as a span's output, not the code that runs inside the with span(...) block. For an override to actually change downstream behavior, your traced function needs to read s.output back out after the block (as in the quickstart above) rather than returning a value it computed before assigning s.output. This is a deliberate simplicity tradeoff — true arbitrary step-skipping would require capturing and replaying closures, which is out of scope for a library this size.

Design notes

• Storage: SQLite by default, one file, no server. Storage is a small wrapper around raw SQL — swap it out if you need Postgres for concurrent writers.
• Nesting: spans nest automatically via contextvars, so this is safe across asyncio tasks (though not yet tested under heavy concurrent load — contributions welcome).
• Serialization: inputs/outputs are stored as JSON via json.dumps(..., default=str), so non-JSON-serializable objects get stringified rather than crashing the tracer.

Async

trace_agent and span both work with async def code — no separate API:

async def retrieve(query):
    async with tracedrift.span("retrieve", input={"query": query}) as s:
        s.output = await my_vector_db.search(query)
    return s.output

@tracedrift.trace_agent(name="qa_agent")
async def run_agent(query):
    docs = await retrieve(query)
    ...

trace_agent detects async def functions automatically and awaits them. Nesting relies on contextvars, which asyncio copies into each Task at creation time — spans nest correctly as long as they run within the same Task as the enclosing trace_agent call; spans started in a separately created Task (e.g. via asyncio.create_task) won't see updates made after that task started.

Cost/token tracking

Record usage on a span with s.set_usage(...) — tracedrift doesn't know anything about specific providers' response shapes, so pull the numbers out yourself and pass them in:

with tracedrift.span("llm_call", input=docs) as s:
    resp = my_llm.complete(docs)
    s.output = resp.text
    s.set_usage(
        prompt_tokens=resp.usage.prompt_tokens,
        completion_tokens=resp.usage.completion_tokens,
        cost_usd=resp.usage.cost_usd,
    )

tracedrift show <trace_id> prints per-span usage and a trace-level total. Programmatically, tracedrift.trace_usage(trace_id) sums prompt/completion/ total tokens and cost across every span in a trace that called set_usage (returns {} if none did).

Web UI

tracedrift serve                 # http://127.0.0.1:8765
tracedrift serve --port 9000

A minimal, read-only, stdlib-only local server: trace list, span tree per trace, diagnosis and usage totals inline. No JS framework, no build step, no auth — it's meant for browsing your own local tracedrift.db during debugging, not for exposing over a network.

What's not here yet

• Framework-specific adapters (LangChain callback handler, DSPy module wrapper)

Contributing

See CONTRIBUTING.md for dev setup, project layout, and design principles the codebase holds itself to. Please also read the Code of Conduct. Found a security issue? See SECURITY.md instead of opening a public issue.

License

MIT