clustertrace 0.9.1

Local-first LLM agent observability — decorator + OpenTelemetry ingestion, traces grouped by execution pattern, cost per call, search, replay, shareable HTML snapshots.

clustertrace

A local-first debugger for LLM agents. Drop in a decorator, run your agent a few hundred times, then look at where it's failing by execution pattern rather than by individual trace.

I built it because I had a multi-step agent that was failing about 20% of the time in random-looking ways, and the only trace-tooling I had was a flat list view that didn't help. Clustering the traces by execution signature turned it from "scroll 80 stack traces" into "two patterns explain ten of the twelve failures, here they are". That's the entire pitch.

In the bundled 60-trace demo, two clusters cover 10 of the 12 failures (83%). Eighteen distinct execution patterns total. You can reproduce the numbers in 30 seconds.

Try it without an API key

pip install clustertrace
clustertrace demo

Loads 60 pre-recorded traces of three agents (research, RAG, tool-use), boots a dashboard on localhost:7777, no API spend. (Pre-PyPI install: pip install "clustertrace @ git+<repo-url>" once the repo's pushed; both URLs go live on first public push.)

If clustertrace demo works on your machine, the rest of this README is just feature surface.

Using it on a real agent

There are three ways to feed traces in. Pick one.

A bare decorator on the functions you want traced:

import clustertrace

@clustertrace.trace(tags={"agent": "research"})
async def plan(query): ...

with clustertrace.span("retrieval", k=5):
    ...

clustertrace.tool_call("web_search", args={"q": query}, result=hits)
clustertrace.tag("user_tier", "pro")
clustertrace.metric("score", 0.85)        # numeric, aggregated to a sparkline

Async-safe. Concurrent asyncio.gather calls produce separate traces; nesting tracks the parent via contextvars.

Or wrap your SDK client, no decorator on your code:

from anthropic import Anthropic, AnthropicBedrock, AnthropicVertex
from openai import OpenAI
import clustertrace

client  = clustertrace.wrap_anthropic(Anthropic())          # direct API
bedrock = clustertrace.wrap_anthropic(AnthropicBedrock())   # AWS Bedrock
vertex  = clustertrace.wrap_anthropic(AnthropicVertex())    # Google Vertex
oai     = clustertrace.wrap_openai(OpenAI())                # OpenAI

It's an explicit wrap, no global monkey-patching. Async clients (AsyncAnthropic, AsyncOpenAI) are detected automatically.

Or just point your existing OpenTelemetry setup at clustertrace as an exporter:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from clustertrace.otel import ClustertraceSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ClustertraceSpanExporter()))

If you already have OTel wired through LangChain, LlamaIndex, Bedrock auto-instrumentation, or your own custom spans, this picks them up. gen_ai.* and llm.* attribute conventions are mapped onto clustertrace's schema, so cost and clustering work on OTel-sourced traces too.

What the dashboard shows

page	content
/	filterable trace list with status/tag/name filters, live polling, per-trace cost
/clusters	distinct execution patterns: count, failure rate, sample trace, longest common failure prefix, top failing nodes
/search	FTS5 search across span name, input, output, error message; supports phrases, OR, NEAR
/metrics	per-metric aggregates and rolling sparklines for whatever you passed to clustertrace.metric()
/failures	per-span error-rate bars, step-of-failure histogram, force-directed call graph
/trace/<id>	Gantt timeline, expandable I/O, tags, metrics, per-span cost

examples/sample-trace.html shows a 16 KB self-contained shareable snapshot: data and renderer embedded in one file, no external assets.

CLI

clustertrace demo                                      # one-step trial with bundled data
clustertrace dashboard                                 # launch local server
clustertrace stats                                     # one-screen DB summary
clustertrace inspect --latest                          # terminal Gantt of last trace
clustertrace inspect --failed                          # most-recent failed trace
clustertrace inspect <trace_id> --expand <span_id>     # dump that span's I/O
clustertrace mcp                                       # MCP server for AI editors (stdio)
clustertrace mcp install --target claude-code          # wire into your editor
clustertrace backfill-cost                             # compute $ for every LLM call retroactively
clustertrace backfill-signatures                       # signatures for older traces
clustertrace snapshot <trace_id> -o trace.html         # self-contained shareable HTML
clustertrace export <trace_id>                         # JSONL to stdout
clustertrace export --all > backup.jsonl               # everything
clustertrace import < backup.jsonl                     # merge (skips existing IDs)
clustertrace replay <trace_id> --entry mod:fn          # re-run with captured args
clustertrace db-path                                   # print SQLite path

MCP server

clustertrace mcp exposes traces, clusters, and search through the Model Context Protocol, so any MCP-capable editor (Claude Code, Cursor, Continue) can ask "show me a failing trace of this pattern" or "diff this trace against a successful one" as a single command.

pip install "clustertrace[mcp]"
clustertrace mcp install --target claude-code   # or cursor, or continue
# restart your editor; the tools appear

Six read-only tools are exposed:

tool	what
list_clusters	distinct execution patterns with count + failure rate
get_trace	full record (trace + spans + tags) for one trace id
search	FTS5 search over span name + I/O + error messages
failure_summary	aggregate failure-pattern view, optionally grouped by tag
recent_failed	the N most recent traces with status=error
compare_traces	structured diff (insert/delete/equal) of two traces' spans

Without --target, clustertrace mcp install prints the JSON snippet for you to paste into your editor's config:

{
  "clustertrace": {
    "command": "clustertrace",
    "args": ["mcp"]
  }
}

v0.9 ships read-only tools only. Annotate/assert mutation tools are slated for v1.0 once I see how the read-only surface gets used in practice.

Configuration

var	default	purpose
CLUSTERTRACE_DB	~/.clustertrace/traces.db	SQLite file path
CLUSTERTRACE_MAX_PAYLOAD_BYTES	32768	per-field cap on serialized span I/O
CLUSTERTRACE_PRICING_JSON	(none)	override or extend the model price table
CLUSTERTRACE_OTLP_MAX_BYTES	16777216	body cap on POST /v1/traces; 413 on overflow

Case study

Maintainer dogfood self-study: a synthetic research agent went from 40% failure to 15% failure after a four-line fix that the cluster page surfaced in about five seconds. Reproducible from examples/case_study_research_agent.py. The doc opens with what it does not prove (no real customer numbers yet); it's a dogfood report, not a testimonial.

Where clustertrace doesn't fit

Production multi-tenant observability with teams, retention policies, PII redaction, and a managed dashboard is a different problem. clustertrace is a debug tool on a single laptop with a single SQLite file. Single-user, no auth, no persistence-tiering. It's intentionally simpler.

FAQ

Why cluster traces instead of just listing them. Even at 60 traces (the bundled demo) the list view doesn't surface the pattern. Clustering collapses them into 18 distinct execution patterns and tells you that 2 patterns cover 10 of the 12 failures (83%). At production volumes that's the difference between reading 1000 traces and reading 2.

Why local-only with no auth. Trade-off: keeps the binary small and the trial frictionless. Single-user is the right default for a debug tool, not a production tracing service.

Does it work with LangChain / LlamaIndex / DSPy. Yes, via OpenTelemetry. Anything emitting OTel spans flows in. gen_ai.* and llm.* attribute conventions are mapped onto the clustertrace schema, so cost and clustering still work.

Streaming responses. The span is logged on completion. Chunk-by-chunk capture isn't implemented yet; v0.5 target.

How deep is the clustering algorithm. Cluster signatures use exact-string equality on a normalised, run-length-collapsed span sequence. Reorderings split clusters today: A→B→C and A→C→B end up as two clusters. Reorder-insensitive matching via set-of-edges or tree-edit-distance is the next algorithmic move. See ARCHITECTURE.md for the full design notes.

How much does the demo cost. Zero. The bundled 60 traces are pre-recorded. The full reproduction script (examples/generate_demo_data.py, 240 traces) costs about $2-3 in Haiku.

Overhead

@clustertrace.trace adds low-microsecond decorator overhead (≈35 µs of pure-Python wrapping on modern hardware), but the SQLite write is the real per-call cost: about 5 ms on Linux/macOS, about 30 ms on Windows NTFS. End-to-end traced-call latency in examples/benchmark.py is dominated by the disk write, not the decorator. For a debug tool on a laptop that's fine; you don't trace 100/sec.

For production:

@clustertrace.trace(sample=0.01)   # log 1% of calls
def hot_path(): ...

@clustertrace.trace(skip=True)     # zero overhead; returns the function unwrapped
def loop_body(): ...

Run python examples/benchmark.py to see the numbers on your hardware.

Known limitations

Streaming responses are logged on completion only, not chunk-by-chunk. The streaming: true attribute is recorded so you can filter on it, but intermediate chunks aren't captured. v0.5 target.

Replay with prompt diff is half-built: clustertrace replay re-runs with captured args, but modifying the prompt before re-invocation isn't yet exposed. v0.5.

Native wrappers only for Anthropic and OpenAI. Bedrock and Vertex work through wrap_anthropic (they share the .messages.create interface). Gemini works through OpenTelemetry.

Single-user, no auth. The dashboard is intended for 127.0.0.1. See SECURITY.md.

Contributing

ARCHITECTURE.md has the design choices; CONTRIBUTING.md has setup and a step-by-step recipe for adding a new SDK wrapper. Real gaps that would meaningfully help users are listed at the bottom of CONTRIBUTING.md.

License

MIT. See LICENSE.