tracesweep

Reads your agent traces. Tells you what's broken. Doesn't touch your code.

An open-source alternative to LangSmith Engine: point it at a corpus of production agent traces, and it screens them, groups failures into named issues, and puts them on a board for a human to rule on.

A sweep is the unit of work — one pass over the corpus. Sweeps are scheduled, compared to each other, and cost a printed number of dollars.

pip install tracesweep          # not yet published

tracesweep ingest ./traces
tracesweep run --budget 5.00    # one sweep
tracesweep issues

Status: pre-alpha. Nothing is built yet. This repo currently holds the name, the design, and the decisions behind it. The blueprint is complete and adversarially reviewed; the code is not written.

Why

LangSmith Engine bills against LangChain-managed inference, and BYOK is explicitly not supported — roughly $750/month at the default org cap. You are paying a vendor's LLM to read your LLM's output, continuously, at exactly the trace volume where it is most useful.

That pricing is structural, and it is the opening. Aggressive representation tiering plus a cheap model at the wide end gets the same job done for single-digit dollars a month on your own keys.

Design in one page

No auto-PR is a security decision, not a scoping shortcut. The system ingests production traces — attacker-controllable end-user text and tool output — into an agent that reads a private repo. Untrusted input, private data, and external write capability in one loop is the dominant threat for this product class. Human review breaks the loop.

What's actually unknown

Nobody publishes precision or false-positive rates for trace issue detection — not LangSmith Engine, not any competitor. That is the one piece of genuine white space, and it is the point of building this in the open.

The base-rate arithmetic is unforgiving: at an implied yield of ~1 issue per 3,000 traces, even 99.9% per-trace specificity produces roughly three false issues for every true one. That is structural, not a tuning problem, and it is what the design is organised around.

Roadmap

• Phase 1 — ingest, compaction, free detectors, screener, a measured recall number on the TRAIL benchmark's 841 annotations, with a cost receipt
• Screener bake-off: Flash-Lite vs Haiku vs GPT-5-nano over TRAIL, recall plotted against cost
• Phase 2 — investigator, verifier, issue board, scheduled sweeps
• Publish precision with a confidence interval — the number nobody else has

License

TBD — see #license.