tracelint

tracelint checks an agent's execution trace for structural defects, locally or in CI. It can read trace files directly or import traces from your existing observability platform.

It reads what a tool-calling agent actually did — the execution trace — and flags structural bugs deterministically, with the exact evidence and a CI exit code. It runs after the run, on the trace, not on your code, and no second model ever judges it. Model-as-judge detection of these defects is unreliable (published trace-error benchmarks show low localization accuracy); many of them are structurally decidable and need no judge — that is the entire premise of the tool.

Website: tracelint.com — what it catches, how it works, and how to wire it into CI. See a live demo report — the constructed validation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious cases) plus the robust-vs-buggy recovery scorecard, generated by tracelint demo.

Quick start — capture → lint → CI

The core loop is the whole product:

run agent in tests → capture the trace → run tracelint → fail CI on provable defects

1. Capture a trace. Most people don't have a trace file yet, so tracelint.capture records one. It wraps your framework's stock OpenInference instrumentation, so the captured file lints with no extra conversion:

pip install "tracelint[capture-smolagents]"   # or [capture-langchain] / [capture-crewai]

from tracelint.capture import capture

with capture("trace.json", framework="smolagents"):
    agent.run("...")            # your agent, unchanged

If your app already instruments the framework (a Phoenix or Langfuse setup), capture leaves that tracing alone and records from the global tracer provider, which phoenix.otel.register() sets. A capture that records nothing raises and says why, instead of writing an empty trace.

2. Lint it.

tracelint check trace.json --format openinference

Exit codes: 0 clean · 1 a finding at a tier you opted into with fail_on · 2 a structurally-provable defect (hard_defect) · 3 an input error: a missing or unreadable file, an empty one, input that doesn't match --format (the error names the format it looks like), or an invalid config, so a misconfigured step never passes having checked nothing. Heuristic candidates never fail CI unless you opt in (fail_on); a suppression (a rule that couldn't run) is disclosed but is not a defect. (A precision study on 200 real agent trajectories found 0/200 false CI failures and that the loop rule doesn't over-fire on retries — and quantifies where the keyless candidate heuristics are noisy on shell traces.)

3. Fail CI on a defect. check returns 2 on a provable defect, so it gates a build directly:

# .github/workflows/agent-tests.yml
      - run: pip install tracelint
      - run: tracelint check trace.json --format openinference   # exit 2 fails the job

See Add to CI for the ready-made GitHub Action, SARIF code-scanning, and pre-commit.

The in-process capture helper covers the three in-process frameworks — smolagents, LangGraph / LangChain, and CrewAI. Langflow is a running product, not a library you call, so it uses the OTel-to-file recipe instead (point its export at a file, then check it). Already collect traces elsewhere? Any supported format works with no capture step.

Prefer to see it work first, with no API key or agent? The keyless demo runs a validation suite and recovery scorecard end to end:

pip install tracelint
tracelint demo --html demo.html

Limitations (read first)

1. Deterministic rules catch structural defects, not whether the final answer was correct.
2. Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven — legitimate value transforms and intentional retries can trip them; each is shown with its evidence for human review, never asserted as a verdict. High-confidence hallucination detection requires the tool schema to declare field origins (x-value-origin).
3. The recovery scorecard needs labeled task outcomes (success oracles); without them it measures behavioral recovery only ("did not crash"), a weaker claim than correctness.
4. A trace is only as complete as its instrumentation. A rule whose required field is missing is suppressed with a stated reason — tracelint never lints a partial trace as if complete. That includes a tool call's arguments: when the trace records them redacted, positionally, or as a bare value, and doesn't hold the model's own call either, they are unknown (never read as {}), and the checks that need them report those calls as not checked.

Supported formats

check reads native tracelint JSON by default; --format points it straight at the trace your stack already emits — no manual schema conversion:

tracelint check trace.json    --format native         # canonical tracelint JSON (default)
tracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL
tracelint check messages.json --format openai          # an OpenAI chat message list
tracelint check trace.json    --format langfuse        # a Langfuse trace export
tracelint check run.json      --format langsmith       # a LangSmith run tree export
tracelint check trajectory.json --format atif          # a Harbor ATIF trajectory (any agent)

OpenTelemetry / OpenInference is a format, not a platform — a set of OTel span conventions. So --format openinference works from any instrumentation that emits them (Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, datasets like TRAIL) with no account, no platform, and no vendor lock — and it's the format the capture helper writes. One shared adapter reaches the whole ecosystem instead of one vendor.

Most rules need no tool schemas, so this works keyless; add --tools tools.json to light up the schema-dependent rules (R1, and R3's high-confidence tier). Don't have one? tracelint init spans.json --format openinference -o tools.json bootstraps a starter contract from the trace — schemas discovered where the telemetry carries them, behavior fields left as placeholders to review. Using MCP tools? tracelint init --from-mcp tools_list.json -o tools.json builds the contract straight from a saved tools/list response: inputSchema becomes the argument schema, and the server's readOnlyHint / idempotentHint become side_effecting / idempotent. Those annotations are advisory, so they are applied only where the server states them — a tool with no hint is left for you to classify rather than guessed. Not sure why rules are suppressing on a trace? tracelint doctor <trace> diagnoses the gap — tool content the instrumentation didn't record vs a missing tools.json — and names the one fix, instead of a wall of suppressions. A multi-trace input (a JSON array, an OTLP export, or a .jsonl file) fans out to one report per run. Spans are grouped into runs by trace id, so a .jsonl file can hold one span per line (Phoenix's to_json(..., lines=True), an OTel file exporter), whole runs, or export batches, and a run split across files is linted whole. From the library:

from tracelint import lint_otel_traces

for report in lint_otel_traces(spans):   # spans: your OpenInference span export (a list of dicts)
    print(report.exit_code)              # 0 or 2, one report per trace

(lint_otel_trace, singular, lints one run's spans and raises if they span several traces.)

The rules

The table below is the one-line summary; docs/rules.md is the full reference — what each rule checks, its tier, and how to resolve it. The text report and the SARIF helpUri link there, one anchor per rule.

hard_event and hard_defect are orthogonal to the finding kind: a tool-error event is a hard_event from a structured signal (the span's or run's own error status, a LangChain ToolMessage with status: "error", or a non-empty error field in the result) but a candidate from a convention or an exception-like string. Fields inside a result, such as status or status_code, are the tool's data: a link checker reports status_code: 404 for a page it checked successfully. They stay candidates until failure_when declares what failure looks like.

R2b follows the data, not repeated text. A failed result often echoes its inputs, so a value counts only when nothing else the agent saw supplied it: an order id the user gave is not from the failure, even when the error repeats it. Retrying the failed tool is handling the error, not misusing it. If a call the value was passed to returns it again (a lookup that may have confirmed it), the use is a candidate. An action that must not run after a failed check, whatever values it uses, is declared instead: requires (R9, see below).

Input format

A native trace is a JSON object (.json, or .jsonl for many):

{
  "run_id": "run-1",
  "steps": [
    {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"},
    {"type": "tool_call", "call_id": "c1", "name": "get_order_status", "args": {"order_id": "4521"}},
    {"type": "tool_result", "call_id": "c1", "content": {"status": "processing"}, "status": "ok"},
    {"type": "tool_call", "call_id": "c2", "name": "cancel_order",
     "args": {"order_id": "4521", "reason": "not_shipped"}}
  ],
  "final": "Order 4521 has been cancelled."
}

tools.json supplies the ground truth the rules check against:

{
  "tools": {
    "cancel_order": {
      "schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                 "required": ["order_id"]},
      "metadata": {"side_effecting": true}
    }
  }
}

A tool can also declare what failure looks like in its result, so a domain failure returned as a transport success (HTTP 200 carrying {"status": "declined"}) is caught structurally instead of slipping through:

{
  "tools": {
    "charge_card": {
      "metadata": {
        "side_effecting": true,
        "failure_when": {"pointer": "/status", "in": ["declined", "failed"]}
      }
    }
  }
}

failure_when is a JSON Pointer into the result plus a match (in / equals / exists); a match is a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A side-effecting tool with no failure_when and an unclassifiable result is suppressed with a reason — never counted as a clean pass.

A tool can also declare what must succeed before it runs (R9). A refund needs a successful lookup of the same order, and a deploy needs a passing pipeline, whether or not the agent copied any value from it:

{
  "tools": {
    "refund_order": {
      "metadata": {
        "side_effecting": true,
        "requires": [{"tool": "get_order", "same": ["order_id"]}]
      }
    }
  }
}

The latest get_order for that order_id must have returned successfully before the refund ran: a retry that passes satisfies it, a later failure doesn't, and a lookup still in flight doesn't count. same (optional) scopes it to one entity, and "succeeded": false asks only that the call returned. A violation is a hard_defect. When the trace can't show whether the lookup succeeded (no error and no failure_when to read), the check is disclosed as not run, never passed. tracelint init proposes requires for calls that followed others.

A contract can also declare sources the run must not reach (R13), beside tools: a benchmark's published solutions, its task repository and mirrors, its leaderboard pages.

{
  "tools": {},
  "off_limits": {
    "sources": ["tbench.ai", "terminal-bench"],
    "reason": "the benchmark publishes each task's solution"
  }
}

A call that names one in a URL (a fetch, a curl) or a search query is a hard_defect, whatever it got back. A source matches as a whole word in any case, with -, _ and a space alike. Its name elsewhere in a call (a task's own data, a file header) is not a request and isn't flagged.

The rules run against one canonical trace schema; a thin adapter translates each source's format into it, so the rules never change: from_openai_messages, from_langfuse_trace, from_langsmith_run, from_otel_spans, and from_atif_trajectory. The adapters are validated against live data, not just the spec — from_otel_spans on real TRAIL benchmark traces, where tracelint deterministically localized real tool errors, a malformed call, and excessive-retry loops with no model in the loop. Real exports vary, so a new source may need a small adapter tweak — and when a field a rule needs is absent, that rule suppresses (says so) rather than guessing, so an unhandled quirk degrades safely.

Integrations & recipes

tracelint reads the telemetry your stack already emits — one shared adapter reaches the whole ecosystem. docs/integrations/ has short, reproducible one-pagers, each validated on a real captured trace:

• Frameworks (capture a trace in a test): smolagents · LangGraph / LangChain · CrewAI — each captures via the helper above and lints via --format openinference with no framework-specific code. Langflow is a running product, so it uses the OTel-to-file recipe instead of the in-process helper.
• Platforms (already have traces?): the always-works baseline is to export the trace to a file and check it — Arize Phoenix · Langfuse · LangSmith · OpenLLMetry / Traceloop · OpenAI / ShareGPT message lists.
• Agent harnesses (benchmark and RL runs): Harbor / ATIF — one reader for every agent Harbor runs (Terminal-Bench and the datasets in its registry).
• Eval suites (already running one?): tracelint drops in as a deterministic, judge-free check — DeepEval (TracelintMetric, via pip install "tracelint[deepeval]") or promptfoo (a Python assertion, no extra to install) — a case passes iff its trace passes the rules.

For Langfuse, pull is convenience sugar on top of that baseline — it fetches a trace straight to a file, so pull → check composes and the file doubles as a saved fixture:

pip install "tracelint[langfuse]"                 # reads your LANGFUSE_* env vars
tracelint langfuse pull <trace-id> -o trace.json   # writes native tracelint JSON
tracelint check trace.json                          # lint it (native is the default format)

These are compatibility validations on real traces, not benchmarks or endorsements.

Straight from a running Arize Phoenix instance, from the library:

from phoenix.client import Client
from tracelint import lint_otel_traces

spans = Client().spans.get_spans_dataframe(project_name="my-agent")
for report in lint_otel_traces(spans.to_dict("records")):   # one report per run
    print(report.exit_code)

Both Phoenix shapes are handled: the span-export JSON (top-level span_kind) and the get_spans_dataframe() records (attributes as attributes.* columns, message lists nested).

Add to CI

tracelint check returns exit 2 on a structurally-provable defect, so it gates a build directly. Point it at the traces your agent test job already produces — a defect fails the job; heuristic candidates never do unless you opt in (fail_on, below).

GitHub Actions — the ready-made action:

name: lint-agent-traces
on: [push, pull_request]
jobs:
  tracelint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... your step that runs the agent and writes traces to ./traces ...
      - uses: AshwinUgale/tracelint@v0.12.0
        with:
          traces: "traces/*.jsonl"
          format: "openinference"     # or native / openai / langfuse / langsmith / atif
          tools: "tools.json"          # optional — lights up R1, R3, R2 predicates

Any CI, without the action — it's one pip install and one command:

pip install tracelint
tracelint check traces/*.jsonl --format openinference --tools tools.json

GitHub code scanning (SARIF) — surface findings in the repo's Security → Code scanning tab and as inline pull-request annotations. --sarif writes a SARIF 2.1.0 file (mapping hard_defect → error, hard_event → warning, candidate → note). Each finding anchors to the line of its first trace step in the uploaded file (line 1 when the step can't be located), so an annotation lands on the relevant record, not the top of the file. It's written before the exit-2 gate, so an if: always() upload step runs even when a defect fails the job:

      - run: tracelint check traces/*.jsonl --format openinference --tools tools.json --sarif tracelint.sarif
      - uses: github/codeql-action/upload-sarif@v3
        if: always()          # upload even when tracelint exits 2 on a hard_defect
        with:
          sarif_file: tracelint.sarif

pre-commit — lint only the trace files a commit touches:

repos:
  - repo: https://github.com/AshwinUgale/tracelint
    rev: v0.12.0
    hooks:
      - id: tracelint
        files: ^traces/.*\.jsonl$
        args: ["--format", "openinference", "--tools", "tools.json"]

One contract for every step — put the settings in [tool.tracelint] in pyproject.toml (or a tracelint.toml), so the Action, pre-commit and any CI command share them. Flags override it.

[tool.tracelint]
format = "openinference"
tools = "tools.json"          # relative to this file
fail_on = "hard_event"        # also fail (exit 1) on hard events; "candidate" for candidates too

[[tool.tracelint.ignore]]     # accept a known finding, with the reason
rule = "R3"
tool = "add_to_cart"
field = "note"
reason = "free-text note the model writes"

Exit codes: 0 clean, 1 a finding at the fail_on tier you opted into, 2 a hard defect, 3 an input or configuration error. An ignore takes a rule and optionally a tool, a field or a path glob; the finding stays in the report with its reason but no longer fails the run, and an ignore that matches nothing prints a warning. A misspelled key or value fails the run (exit 3) instead of quietly loosening the gate. The nearest config file wins, searched up to the repository root; --config FILE names one.

Adopt it on traces that already have findings — record a baseline once and commit it; CI then fails only on what's new:

tracelint check traces/*.jsonl --baseline tracelint-baseline.json --update-baseline   # once
tracelint check traces/*.jsonl --baseline tracelint-baseline.json                     # in CI

(or baseline = "tracelint-baseline.json" in the config). A finding is matched by its trace file, its rule and the tools and fields it involves, never by step position or value, so a re-run of the agent still matches. The baseline accepts as many of each as it recorded: a new kind, one more of the same, or a worse tier fails. It also records what each rule could check, so a run that checks less fails too (exit 1): a tool a rule can no longer check (content capture switched off), or a rule that now checks nothing (schemas deleted). ratchet = false turns that part off. Accepted findings stay in the report, marked "in the baseline", and ones that no longer occur are noted so you can tighten the file with --update-baseline.

Library

from tracelint import ToolRegistry, default_rules, lint_trace, load_source

registry = ToolRegistry.load("tools.json")
for trace in load_source("trace.json", "openinference"):   # or native / openai / langfuse / langsmith
    report = lint_trace(trace, default_rules(), registry)
    print(report.exit_code)      # 0 or 2
    for f in report.active_findings:
        print(f.rule, f.tier.value, f.summary)

load_source returns every run in the file (a span export can hold several), in any supported format. Trace.load reads only tracelint's native trace format.

See examples/lint_openinference_phoenix.py for an offline, keyless end-to-end run (Phoenix-shaped spans → findings, with and without a tool registry).

Advanced

These layers sit around the core loop and are opt-in.

Native test assertion (pytest). Opt in with pytest_plugins = ["tracelint.pytest_plugin"] in your conftest.py; the trace_capture fixture then captures a run and lints it inline, so a hard defect fails the test (candidates never do):

def test_agent(trace_capture):
    with trace_capture(framework="smolagents"):
        agent.run("refund order A100")   # a hard defect in the trace fails this test

Pass assert_clean=False to inspect cap.report yourself instead of auto-failing (cap.reports has one report per run when the block runs the agent more than once). A capture with nothing to lint fails the test rather than passing as clean. It's a thin wrapper over the capture helper, so the same capture-<framework> extra applies.

Recovery scorecard & fault injection. Measure how an agent behaves under injected faults, scored against deterministic success oracles:

tracelint scorecard --demo --faults timeout,error,rate_limit --runs 5

The baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to behavioral recovery, labeled as weaker.

Write findings back into Langfuse. Beyond reading a trace, tracelint can write its verdict back into Langfuse's own Score model so it appears beside your normal evals. This is the one part that overlaps with what a platform does itself, so it's an advanced recipe, not the primary path — the baseline (export or pull a trace, then check it) always works without it:

pip install "tracelint[langfuse]"        # v3 SDK; reads your LANGFUSE_* env vars

tracelint langfuse check --trace <trace-id> --tools tools.json               # read-only: prints the plan
tracelint langfuse check --trace <trace-id> --tools tools.json --write-back  # writes Scores

With --write-back, tracelint writes tracelint.passed (BOOLEAN) and tracelint.hard_defects (NUMERIC) at the trace level, plus one score per certain finding attached to the exact offending observation. Heuristic candidates are shown in the terminal but never written back. Scores are keyed by a stable finding id, so re-running updates them in place rather than duplicating.

Development

python -m pytest
ruff check src tests

The core is dependency-light (jsonschema + stdlib) and the whole test suite is deterministic and offline. A real OpenAI trace-generating agent lives behind the opt-in [real-agent] extra and is never part of the linter. Python 3.10–3.12.