tracegraph

Causal-graph analysis of LangGraph agent traces — time-travel debugging and root-cause analysis on a causal graph.

tracegraph ingests agent execution traces that already exist (LangGraph checkpoint, Phoenix CLI exports, and OpenInference/OTLP spans) into a normalized causal graph, then runs analyses the mainstream observability tools don't:

• explain — walk backward from a failure over the raw causal graph to every real cause.
• diff — exact structural regression diff between two runs (rooted-tree / AHU isomorphism).
• cross-trace pattern matching — e.g. "tool X → retry → tool X → failure" across all runs.

It is not a checkpointer — a graph-backed BaseCheckpointSaver already exists. tracegraph is the read-path analysis layer. See docs/FEASIBILITY.md for the full rationale and competitive landscape. Korean readers can start with docs/USAGE_KO.md, or run the hands-on notebook examples/getting_started_ko.ipynb from a repository checkout. Current delivery evidence, known limitations, and next work are maintained in docs/HANDOFF.md.

Install

uv tool install agent-tracegraph
# or: python -m pip install agent-tracegraph

# With the optional Cypher accelerator (LadybugDB):
uv tool install "agent-tracegraph[cypher]"

# Or run it without installing:
uvx --from agent-tracegraph tracegraph --version

The distribution is agent-tracegraph; the command and the import package are tracegraph. The plain name was already taken on PyPI by an unrelated project, so only the name you install differs — you still run tracegraph and still import tracegraph.

Requires Python 3.12. CI exercises 3.12 on Linux; other versions and platforms are untested rather than unsupported.

Design commitment: causality is never faked

We persist the raw multi-parent causal graph as the system of record (CAUSED_BY, keeping every real predecessor) and treat the single-parent tree (TREE_PARENT) as an explicitly derived, lossy projection. Any step whose causality is collapsed for the tree view is flagged projection_lossy, so RCA never silently presents a fabricated cause.

Usage

# ingest a LangGraph thread's checkpoint history from a SqliteSaver DB
tracegraph ingest --sqlite trace.db --thread A -o A.json
tracegraph ingest --sqlite trace.db --thread B -o B.json

tracegraph inspect A.json            # render the causal tree (errors in red)
tracegraph explain A.json <step_id>  # raw causal chain back to the root cause
tracegraph diff A.json B.json        # structural regression diff (AHU isomorphism)
tracegraph validate A.json traces/   # validate artifacts before CI/query jobs
tracegraph analyze A.json            # automatically select and explain failures
tracegraph check B.json --baseline A.json --rule failures-not-increased   # regression verdict for CI

# Phoenix: one command after configuring the official `px` CLI
tracegraph phoenix doctor
tracegraph phoenix diagnose                 # latest failed trace; latest trace as fallback
tracegraph phoenix diagnose <trace-id>      # explicit trace
tracegraph phoenix diagnose --project my-agent
# save only the body-free normalized evidence/report when needed
tracegraph phoenix diagnose <trace-id> --save-artifact safe.json --json-out report.json

tracegraph presets                   # list cross-trace query patterns
tracegraph query tool-failure A.json B.json   # find a causal pattern across many traces
tracegraph query tool-retry-failure *.json    # explicit retry marker, then same tool failing
tracegraph query tool-failure --backend ladybug A.json B.json   # use the optional Cypher accelerator

# export versioned, body-free governance evidence (an empty match set is a valid report)
tracegraph export-review-candidates tool-retry-failure *.json -o candidates.json

# export an allowlisted incident post-mortem (Markdown) for a long-term memory store
tracegraph export-incident-memo A.json -o incident.md

Frequently-used options, beyond --out / --json-out:

Regression check

check judges one current run against one baseline you name, with rules you name, and answers with an exit code a CI job can use directly:

tracegraph check current.json --baseline baseline.json \
  --rule failures-not-increased --rule pattern-not-increased:tool-retry-failure \
  --json-out check.json

Rules: failures-not-increased (recorded errors per step (kind, name) and in total), pattern-not-increased:<preset> (confirmed matches of a preset), and topology-unchanged (derived tree with labels) or topology-unchanged:shape. Counts compare step signatures, because no source records which step of one run is which step of another; two same-named steps trading which one fails is not detected (analyze --baseline shows the positional view). Evidence that is unknown is never read as zero. When something the selected rule depends on is unknown — an unset status, a value the adapter substituted (unknown_fields) or a missing name on a step the rule could count, or a capture gap that may have left a step out of the artifact — it blocks PASS on the current side and FAIL on the baseline side, so the rule reports INSUFFICIENT_EVIDENCE. The topology rule compares trees rather than counts, so such a gap on either side makes it insufficient even when the observed trees match. Unknowns a rule cannot count leave it decided: an unset CHAIN does not block pattern-not-increased:tool-failure, and a withheld error message blocks only timeout. Pairs from different sources or ingested before capture provenance (artifact version 7) are insufficient for every rule; pairs read with different causal settings, or with causal edges chosen heuristically at ingest, are insufficient for the rules that read causal structure (multi-step presets and topology). A step without a name is treated as unknown wherever a rule compares names, and LangGraph checkpoint names are inferred, so the labeled topology rule is insufficient for LangGraph runs; the shape rule is not. check takes exactly one current and one baseline input (artifact, OTLP export, Phoenix export or -), never a directory, and never picks a baseline. It also assumes each input is a complete record of its run: an export missing a failing span cannot be told from a clean one. The --json-out report is tracegraph.check v1 (contracts/check-report.schema.json), bound to both canonical artifact digests and each rule's id, version and parameters.

Phoenix-first workflow

Phoenix remains the trace UI, evaluation, and operational-observability layer. tracegraph adds deterministic causal diagnosis, retry-pattern detection, and baseline regression analysis without copying prompt or output bodies into its artifacts.

Instrument LangGraph with the official OpenInference integration and configure Phoenix CLI 1.0.4 or newer as described in the Phoenix LangGraph guide and Phoenix CLI reference. Check the connection, then diagnose without finding a trace id first:

tracegraph phoenix doctor
tracegraph phoenix diagnose
tracegraph phoenix diagnose --project my-agent
tracegraph phoenix diagnose <trace-id>
tracegraph phoenix diagnose <trace-id> --baseline <known-good-trace-id>

Without a trace id, tracegraph scans Phoenix's 20 newest traces and selects the newest failed trace. If none failed, it says so and diagnoses the newest trace, so a first successful run is still useful. Explicit ids bypass that selection. The command invokes only read-only px trace list/get --include-annotations exports, analyzes them in memory, and does not persist the raw Phoenix response. Phoenix connection details and credentials remain owned by px profiles or environment variables; tracegraph never accepts an API key option. For an already exported file or a shell pipeline:

Automatic retry diagnosis intentionally requires an upstream retry: CHAIN marker. SyncMill's controlled retry telemetry emits this contract; generic OTLP and LangGraph-checkpoint producers that do not emit it will not be labeled as retries automatically. Their repeated-tool signal remains available with tracegraph query tool-repeat-failure-heuristic, but this inference-only preset cannot create governance review candidates.

tracegraph ingest-phoenix --file phoenix-trace.json --out tracegraph.json
tracegraph analyze phoenix-trace.json --json-out analysis.json
px trace get <trace-id> --format raw --no-progress | tracegraph analyze -

Phoenix exports preserve span parents but currently do not expose the original OTLP span links. These artifacts are explicitly marked causal_fidelity=parent_only; tracegraph warns that additional fan-in causes may be missing instead of claiming a complete DAG.

The exact Phoenix CLI and server versions used by CI are centrally pinned. See the Phoenix runtime pin maintenance runbook for the controlled upgrade and contract-verification procedure, and the Phoenix + SyncMill operational E2E runbook for the scheduled real-server verification boundary.

The default safe-v1 privacy contract governs analysis reports (analyze, phoenix diagnose, --json-out). It retains structural IDs, identifier-shaped operation names, kind/status/time, tokens, explicit cost, and annotation name/label/score. It drops prompts, inputs/outputs, messages, tool arguments/results, retrieved documents, arbitrary metadata, raw errors and stacktraces, annotation explanations, session/user/project identifiers, and credentials. Missing metrics are reported as unavailable, never as zero.

Artifacts are not reports. Phoenix ingestion applies the same filtering at the source (identifier-shaped span names only, error messages dropped), but the OTLP and LangGraph adapters record the span name and error text they were given: OTLP keeps status.message and exception messages, and LangGraph stringifies the configured error state channel. Those values reach the artifact on disk and are printed verbatim by inspect and explain. If your producer puts prompt text or secrets in span names or error strings, treat artifacts from those two adapters as sensitive, or ingest through Phoenix. See "Analysis and output guarantees" below.

For link-preserving analysis, use the optional Collector fan-out example at examples/otel-collector-phoenix-tracegraph.yaml. It sends the normal stream to Phoenix and an allowlisted JSONL copy to tracegraph. Then run:

tracegraph analyze /tmp/tracegraph-otlp.jsonl --trace <trace-id>             # diagnose in memory
tracegraph ingest-otlp --file /tmp/tracegraph-otlp.jsonl --trace <trace-id> -o full-dag.json
tracegraph analyze full-dag.json

analyze reads an OTLP span export (JSON or Collector JSONL) directly, with the same adapter settings as ingest-otlp, so the report is byte-identical to analysing the ingested artifact; nothing is written unless --json-out asks for a report. The input is identified by shape alone (schema_version → artifact, resourceSpans → OTLP, spans/traces/data → Phoenix); a payload carrying both OTLP and Phoenix records is refused, naming the explicit ingest command for each format (or, for a list, JSONL or envelope, asking you to extract one format's records). Already have an export? The shortest offline first run is tracegraph analyze your-export.otlp.json (add --trace <id> when the file holds several), then copy the explain command it prints after saving the artifact it names. The artifact caveat above still applies: ingest-otlp keeps error text, reports stay safe-v1.

The JSON analysis-report contract is versioned at contracts/analysis-report.schema.json. Schema v2 can also carry body-free Toolgraph preflight evidence: an exact SHA-256 artifact digest, non-negative graph generation, and bounded verdict. This is trace metadata, never a CAUSED_BY edge.

$ tracegraph inspect A.json
A (langgraph)
└── -1: CHAIN
    └── 0: CHAIN
        └── 1: plan
            └── 2: call_tool  ✗ tool failed on input 'boom-please'
                └── 3: handle_error
                    └── 4: respond
6 steps · 1 tool · 1 error · 0 lossy-projection

$ tracegraph diff A.json B.json
NOT IDENTICAL
  • only in A under CHAIN > CHAIN > plan > call_tool: subtree handle_error(respond)
  • only in B under CHAIN > CHAIN > plan > call_tool: subtree respond

$ tracegraph query tool-failure A.json B.json
A: call_tool
1 match(es) across 1 trace(s)

$ tracegraph query tool-retry-failure retry.json
retry: search → retry:search → search
1 match(es) across 1 trace(s)

The official tool-retry-failure@v2 matches three consecutive causal steps: TOOL X → CHAIN retry:* → TOOL X (ERROR). The producer must supply the explicit retry marker and the endpoint must name the same tool. This strict pattern is review-exportable. tool-repeat-failure-heuristic uses an unbounded gap and tool-retry-failure-near a bounded gap (up to 30); both are query-only heuristics. The generic matcher still supports gaps and back-references. Unbounded or oversized Cypher patterns fall back to the Python matcher without truncating results.

Every shipped preset has a stable pattern_id and positive integer pattern_version, shown as pattern-id@vN by presets and query. Increment the version when match semantics or review-export eligibility changes; wording-only description edits keep the current version.

export-review-candidates accepts the same artifact-file/directory batches as query and writes a deterministic schema-v1 envelope. Only presets ending in a failing TOOL predicate are eligible. The matched endpoint name must already be a qualified server::tool key — tracegraph never guesses that identity. Each candidate contains only run_id, pattern id/version, tool key, and the SHA-256 digest of the exact normalized artifact that was queried. There are no prompts, outputs, errors, step IDs, trace IDs, or local paths, and the command never changes a Toolgraph manifest or policy. A matching artifact without run_id fails the entire batch and leaves an existing output untouched; export never silently drops evidence. A *.json file in a directory that is not a valid artifact is skipped with a warning, as for query; pass --strict to fail the batch instead. The public schema is a repository-vendored contract (not wheel package data) at contracts/review-candidates.schema.json; G3 consumers should vendor that file explicitly.

SyncMill can consume the report without importing this package:

SYNCMILL_BOARD__ENABLED=true syncmill board import-review-candidates candidates.json

The importer creates deterministic, human-required review items. It does not run agents, invoke Toolgraph, or reinterpret strict enforcement policy. Controlled SyncMill E2E covers qualified tool spans, explicit retry causality, Phoenix streaming, and review intake. Fixture tests remain separate evidence; general deployment coverage and production acceptance are not established by the controlled workflow.

Toolgraph G3 can independently review the same report with review-candidates list/annotate. Like SyncMill, it derives the same UUIDv5 from the exact candidate tuple for correlation, but stores dispositions in a separate exact-report-digest-bound sidecar. Toolgraph accepted neither completes the SyncMill board item nor changes a manifest, selector result, blast radius, preflight result, or graph state.

Status

Alpha (0.x): the artifact, report and review-candidate formats are versioned contracts, and any change to them is called out in the changelog.

Known limits:

• A LangGraph task that was scheduled but recorded nothing (a run stopped by the recursion limit, killed, or held at an interrupt_before/interrupt_after breakpoint) leaves no write. It is reported only when ingest --graph names the compiled graph (below); without it such a run reads ok unless some other step recorded an error. Trace status stays ok either way. A task paused by interrupt() is reported without a graph. Both are reported only at the root graph: the subgraph node is named, not the node inside it.
• Phoenix exports carry no span links, so their artifacts are parent-only; additional fan-in causes may be missing.
• Report privacy (safe-v1) applies to reports; OTLP and LangGraph artifacts keep the producer's names and error messages.
• CI covers Python 3.12 on Linux.

Caveat for the checkpoint adapter: it is a checkpoint-level view (one node per super-step), and node names/kinds are best-effort metadata. Phoenix/OTLP adapters are span-level. Subgraph checkpoints are namespaced, but cross-namespace causality remains limited to parent links declared by LangGraph checkpoint metadata.

Develop

uv sync                    # core only
uv sync --extra cypher     # include the optional LadybugDB backend
uv run pytest              # headless

CI also builds the wheel and installs it with locked runtime dependencies in fresh core and Cypher environments. scripts/verify_wheel.py exercises the installed CLI outside the checkout, checks candidate golden bytes and baseline diagnosis, and verifies the optional-backend boundary and parity. See the handoff runbook for local reproduction and the distinction between local checks and remote/server evidence.

Releases are tag-driven: a v* tag publishes to PyPI through Trusted Publishing, and a test-v* tag rehearses the same build against TestPyPI. The procedure, including what is verified before anything is uploaded, is in the release runbook.

The optional Cypher accelerator (agent-tracegraph[cypher], LadybugDB) is not required for the core; explain and query can opt into it with --backend ladybug. Its tests are marked @pytest.mark.cypher and skip cleanly without the extra. The tested LadybugDB compatibility range is declared in pyproject.toml; its embedded cache format is not load-bearing because the portable JSON artifact remains authoritative and can rebuild the cache.

Analysis and output guarantees

Implicit ingest filenames keep short portable identifiers as <id>.json; other IDs use trace-<sha256>.json. Existing default outputs cause exit 2; use --out to explicitly replace a file. Outputs cannot alias an input or another output. Completed artifacts are published atomically. SQLite ingest uses a read-only connection and a private backup with a 30-second deadline, preserving source data, schema, and journal mode. Live WAL readers can still participate in SQLite's WAL/shared-memory coordination.

safe-v1 reports retain identifier-shaped display names and replace other display text with deterministic SHA-256 aliases, and disclose in warnings when an alias actually reached the report. Trace/step identifiers remain; this is not anonymization. Raw artifact names and bytes are not rewritten for report privacy or matching, so the artifact — unlike the report — can still carry whatever the producer put in a span name or error message. Phoenix ingestion is the exception: it filters at the source, keeping span names and annotation names and labels only when they are identifier-shaped, and dropping status messages. Identifier-shaped is not the same as anonymous: an e-mail address passes. export-review-candidates output is not passed through the report redactor: its tool_key is the raw endpoint span name. export-incident-memo goes further than safe-v1: it writes no step names, error messages or evaluation labels at all. Steps appear as kind-based aliases and the run id as a truncated digest; the rest is structural metadata such as trace status, causal fidelity, step and error counts, preset pattern ids and versions, edge origins, fidelity disclosures, and the digest of the exact artifact bytes. Logical comparison keys are JSON-encoded typed paths; consumers should treat them as opaque strings. Containment and unknown edges preserve all error investigation candidates, deeper first. Explicit causal ancestry supplies context, not proof that an exception propagated. Explicit OTLP OK takes precedence over handled exception events on new ingestion; old artifacts are not reinterpreted. A span whose status is UNSET and that carries no exception event (one with a message or type makes it error in both importers) is read differently by source. ingest-otlp records it as ok, because UNSET is OpenTelemetry's default status and producers such as SyncMill leave successful spans unset. Phoenix ingestion keeps it as unset. Trace status is the same either way, but step statuses differ wherever they are shown, and a preset that requires an ok step does not match an unset one: a SyncMill run where an agent attempt left UNSET directly causes a failing gate: tool reports gate-failure-after-success through ingest-otlp and not through Phoenix. Compare a run with a baseline from the same source; mixing them can report a step whose status differs only by source as a behavior change (ok to unset, or the reverse).

LangGraph ingestion reads both the configured error state channel and native task failures from checkpoint pending writes. A checkpoint is an error where the channel first holds an error. A node that writes the channel again while it still holds one may have failed again, or only have returned the whole state (and a reducer can change the value either way); the checkpoint cannot tell which, so such a write is not recorded as a failure but disclosed as the capture gap langgraph.rewritten_error_channel. That makes check's count rules one-sided and success-dependent patterns such as gate-failure-after-success insufficient either way. Clear a plain (last-value) error key before retrying ({"error": None}) to have every attempt recorded; a reducer on the channel decides whether None clears it. A node that raises becomes a derived task step under the checkpoint that scheduled it, never a mark on that checkpoint, whose own name belongs to the node that produced it. The failing node is named by recomputing LangGraph's task id from checkpoint data; when nothing proves which node ran, the step stays unnamed rather than borrowing a neighbour's name. When one task raises, LangGraph cancels its siblings and records that through the same channel; a cancelled task is reported as an unset step with no message, because it was torn down rather than at fault. Only the latest persisted error survives per task, so repeated attempts are not reconstructed, and Send packets are nameable only where langgraph.types is importable — an ordinary install discloses them instead. A task that called interrupt() and whose only write on the thread's newest root checkpoint is that interrupt becomes an unset step with source interrupt under that checkpoint: the recorded history holds no resume and no result for it. That describes the history, not whether LangGraph would resume the task today. An interrupt recorded together with a resume or an error and no result is ambiguous in the checkpoint itself — under LangGraph 1.2.2 a task resumed with nothing to write looks exactly like one paused a second time — so it is disclosed at ingest, not reported. Scheduled tasks need the compiled graph: tracegraph ingest --sqlite trace.db --thread A --graph path/to/agent.py:graph (the langgraph.json form) or --graph pkg.module:graph. LangGraph's own get_state then lists the tasks that graph schedules at the thread's newest root checkpoint, and each one with no write there becomes an unset step with source scheduled. Whether a task finished, failed or paused is still read from the checkpoint's writes, so a scheduled step never duplicates a task or interrupt step. A scheduled step does not claim that the task never started: a task killed mid-run records nothing either. The answer belongs to the graph you pass: a different graph can schedule different tasks from the same checkpoint, and nothing in one checkpoint proves which graph wrote it, so the artifact records the graph's structural fingerprint (trace.schedule_graph) — even when nothing is scheduled, which is how "checked, nothing pending" differs from "not checked". A Command(goto=...) input saved on that checkpoint but not yet applied, and Send packets addressed to a node the graph lacks, are left out of LangGraph's own answer; ingest discloses both instead. --graph runs code: it imports the module (top-level code runs), and reading the checkpoint runs the graph's state callbacks — schema mappers and validators, reducers, custom channels, managed values, cache-key functions — though no node runs. It is never inferred. Every ingest records what the adapter could not observe: trace.capture is an object whose gaps list holds a fixed code and count per ingest disclosure (empty when there was none), and a step's unknown_fields names a kind, name or error message the adapter substituted. That record makes a fresh artifact declare schema_version 7. The envelope otherwise follows content, which is what an artifact written by an earlier release keeps when it is loaded and saved again: a graph read (--graph) needs 6, interrupt steps 5, a derived tree that relies on EdgeOrigin precedence 4, derived task steps 3, and anything else 2, byte for byte (a v1 file is migrated to 2 as before). Missing observed errors still do not prove success: an artifact ingested by an older build, or one using an unrecognized checkpoint layout, carries no derived step and looks exactly like a clean run, which is why the capture disclosure is unconditional. Link preservation means valid in-trace links only: declared_dag and links_preserved say how links were read, not that every declared link resolved. A link that names this trace but no span in the export, names no trace, or names another trace is not recorded as a cause; each kind is disclosed on stderr and recorded as a capture gap (otlp.unresolved_span_link, otlp.unscoped_span_link, otlp.foreign_trace_link), and check treats the first two like a step that may be missing. Structural diff compares the derived tree and discloses lossy step counts. Metrics sum available span observations: partial coverage and producer aggregation can undercount or double count. Coverage gaps are disclosed rather than absorbed — a wall duration derived from a subset of steps says so, one whose end precedes its start is withheld, and a single unparseable cost withholds the whole total instead of understating it. An OTLP or Phoenix import records an unreadable total cost as unreadable for that reason, and reports token counts, other costs, currencies and evaluation scores it cannot read as unavailable, disclosing each on stderr; the producer's value is never kept. File input, stdin and Phoenix px output share one size limit (TRACEGRAPH_MAX_ARTIFACT_BYTES). query reads artifacts one after another and keeps only the matches it prints (plus the traces --explain renders). With --limit, its peak memory therefore grows with the number of artifacts only by a small per-file record (path, trace id); without it, by the matches it prints. It still enumerates every match of the trace it is reading, to report the exact total: one trace with very many matches is not bounded by --limit.

Both stores isolate returned mutable objects. InMemory node upsert replaces by ID; Ladybug's existing insertion semantics are unchanged. Use is_isomorphic or diff for deep-tree equality; canonical() preserves nested tuples, whose external Python equality can still reach the interpreter recursion limit. CI currently covers Python 3.12/Linux; other supported Python versions and platforms require separate validation.

Contributing, security, and license

• Contributing: CONTRIBUTING.md — development setup, what this project treats as a defect, and the Contributor License Agreement.
• CLA: CLA.md. Signing is one-time per GitHub account per repository; a workflow comments on your first pull request with instructions.
• Code of conduct: CODE_OF_CONDUCT.md — applies to issues, pull requests and discussions.
• Security: SECURITY.md — please use private vulnerability reporting rather than a public issue. Read the redaction boundary there before sharing artifacts: reports are filtered, artifacts are not.
• License: Apache License 2.0 — see LICENSE. Copyright 2025-2026 DAPADA Inc. and memtomem contributors.