Traceport Agent

Traceport Agent is an open-source, file-based improvement loop for production AI agents. It reads telemetry through Traceport, screens compact sessions, investigates recurring failures with LangChain Deep Agents, and produces reviewable issues, recurrence evaluators, regression assertions, and coding-agent handoffs.

How it works

flowchart TD
    J["External scheduler or manual run"] --> F["Fetch sessions through Traceport"]
    C["Watermark + session fingerprints"] --> N{"New or changed session?"}
    F --> N
    N -- "No" --> Z["Finish scan + advance cursor"]
    N -- "Yes" --> V["Build bounded, redacted transcripts<br/>with metrics and objective telemetry facts"]
    V --> B["Group into batches of N<br/>default: 20, concurrency: 4"]
    O["Approved Agent Overview<br/>+ optional agent description"] --> S
    B --> S["Agentic screening<br/>across each bounded session"]
    S --> D{"Candidate failure?"}
    D -- "No" --> Z
    D -- "Yes" --> T["Write all fetched traces once<br/>to a temporary redacted workspace"]
    T --> I["Focused investigation<br/>larger bounded transcript + targeted grep"]
    R["Optional read-only repository"] --> I
    I --> M["Diagnose, cluster, and compare with<br/>open, resolved, and dismissed issues"]
    M --> E{"Evidence threshold met?"}
    E -- "No" --> Z
    E -- "Yes" --> Q["Create, update, or reopen issue"]
    Q --> A["Test recurrence evaluator<br/>against evidence + clean controls"]
    A --> G["Create redacted<br/>regression assertions"]
    G --> H["Optional auditable<br/>coding-agent handoff"]
    H --> Z

Finding categories

Every candidate and durable issue uses one category from this stable taxonomy:

AI agents

The Python host orchestrates these two agents directly. There is no supervisor LLM that delegates or reformats their responses.

Evaluator testing and regression assertion creation are separate host-controlled pipeline steps, not AI agents. This keeps artifact generation predictable and prevents a model from inventing executable evaluator logic. Semantic recurrence evaluators receive only trace metrics, compact span summaries, and bounded beginning/end transcript excerpts. Full span inputs, outputs, raw provider records, and temporary telemetry files are never dumped into the evaluator prompt.

The first run drafts the Agent Overview and pauses for approval. Later one-shot runs process only new or changed sessions. Each session is screened once per fingerprint, batches keep model calls efficient, and only flagged sessions receive the larger investigation context. The loop persists reviewable artifacts rather than modifying the traced application.

V0.1 is deliberately advisory:

• no trace-provider writes
• no web UI or database
• no repository writes or code execution
• no generated Python or JavaScript evaluators
• no PR creation or deployment
• no external handoff delivery

Install

Install the provider extra you need:

uv add 'traceport-agent[langsmith]'
uv add 'traceport-agent[logfire]'
uv add 'traceport-agent[langfuse]'
uv add 'traceport-agent[phoenix]'

Python 3.11 or newer is required. Credentials must be supplied through provider and model environment variables; they are never accepted in traceport-agent.toml. The CLI automatically loads a .env file from the current directory. Copy .env.example to .env; .env is ignored by Git and excluded from package artifacts.

LangSmith connection example

Install Traceport Agent with the LangSmith adapter:

uv add 'traceport-agent[langsmith]'

Export the LangSmith read credential and the credential for the model Traceport Agent will use. The values below are placeholders; do not put them in traceport-agent.toml or commit them:

export LANGSMITH_API_KEY="lsv2_..."
export OPENAI_API_KEY="sk-..."

To use GPT-5.6 Luna through Pydantic AI Gateway instead of a direct OpenAI key, use the Gateway token with its OpenAI-compatible chat route:

export OPENAI_API_KEY="pylf_v..."
export OPENAI_BASE_URL="https://gateway-us.pydantic.dev/proxy/chat/"

Then configure the model as openai:gpt-5.6-luna. The Gateway's Anthropic proxy route is only for Anthropic-protocol models; it is not the route used for Luna. See the Pydantic AI Gateway integration documentation.

Hosted LangSmith uses its default API endpoint. Set these only when your LangSmith deployment or account requires them:

export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_WORKSPACE_ID="your-workspace-id"

Before configuring the agent, you can verify the Traceport connection directly. my-agent must be the exact LangSmith project containing the traces you want to analyze:

traceport traces list \
  --provider langsmith \
  --project my-agent \
  --since 7d \
  --limit 5

Then initialize Traceport Agent with the same provider and project:

traceport-agent init \
  --provider langsmith \
  --project my-agent \
  --repo https://github.com/example/my-agent \
  --model openai:gpt-5.6-luna

traceport-agent doctor

This writes the provider and project to traceport-agent.toml, while credentials remain in the process environment. doctor checks that the LangSmith adapter, model, state directory, and optional repository are reachable without printing credential values. LANGSMITH_PROJECT is not required for reading: Traceport Agent passes [trace].project from its TOML configuration to Traceport.

Optional self-tracing

Traceport Agent can trace its own screening, investigation, evaluator, and issue-lifecycle work to a separate LangSmith project. This is distinct from the source telemetry that it analyzes:

• Source telemetry uses [trace].project plus LANGSMITH_API_KEY, LANGSMITH_ENDPOINT, and LANGSMITH_WORKSPACE_ID.
• Traceport Agent telemetry uses only the four TRACEPORT_AGENT_LANGSMITH_* variables below.

Set all four Agent variables, or leave all four unset:

export TRACEPORT_AGENT_LANGSMITH_API_KEY="lsv2_agent_..."
export TRACEPORT_AGENT_LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export TRACEPORT_AGENT_LANGSMITH_WORKSPACE_ID="agent-observability-workspace-id"
export TRACEPORT_AGENT_LANGSMITH_PROJECT="traceport-agent-observability"

The Agent tracing client never falls back to the unprefixed source variables. Ordinary LANGSMITH_TRACING or LANGSMITH_PROJECT settings also cannot implicitly enable Agent tracing. The same LangSmith account may be used, but the endpoint, workspace, project, and key remain explicit. Traceport Agent rejects a self-tracing destination that resolves to the same endpoint, workspace, and project as its LangSmith source.

When configured, each run appears as a traceport-agent.scan trace with child spans for fetching, concurrent screening, investigation, evaluator work, and issue updates. Agent model calls remain nested under those operations. Metadata contains operational identifiers and counts, never either API key. Run traceport-agent doctor to see self-tracing: disabled or a credential-free configured status.

Quick start

traceport-agent init \
  --provider langsmith \
  --project my-agent \
  --repo https://github.com/example/my-agent \
  --model openai:gpt-5.6-luna

traceport-agent doctor
traceport sessions list --provider langsmith --project my-agent --since 7d --limit 5
traceport sessions show SESSION_ID --provider langsmith --project my-agent
traceport-agent run --verbose
traceport-agent overview show
traceport-agent overview approve
traceport-agent run --verbose
traceport-agent issues list

The first run drafts state/memories/AGENTS.md and exits. Analysis begins only after the exact current file hash is approved.

By default the CLI reads traceport-agent.toml and stores state under .traceport-agent/. Override them before the command:

traceport-agent --config /config/traceport-agent.toml --state /state run

Run this one-shot command from cron, Kubernetes, Cloud Run Jobs, or another external scheduler. A six-hour cadence is a useful starting point.

Commands

traceport-agent init --provider ... --project ... --repo ... --model ...
traceport-agent doctor
traceport-agent overview show
traceport-agent overview approve
traceport-agent run [--since ...] [--until ...] [--max-sessions ...] [--verbose]
traceport-agent issues list|show|close|dismiss|reopen|prioritize|send
traceport-agent evals list|show|test

issues send invokes LogDispatcher. Its result is logged or already_logged; it makes no network call and never reports that the handoff was sent.

Use --verbose to print safe, immediately flushed progress while a scan is running:

traceport-agent run --max-sessions 10 --verbose

It reports fetch counts, new or unchanged sessions, batch progress, candidate investigations, issue actions, evaluator testing, regression assertion creation, and the final summary. It never prints transcript content, model prompts, diagnoses, credentials, or repository contents.

Trace and session inspection belongs to the base Traceport CLI and requires no investigator model call. Every truncated message preserves both its beginning and end by default:

traceport traces list --provider langsmith --project my-agent --since 30d
traceport traces show TRACE_ID --provider langsmith --project my-agent
traceport sessions list --provider langsmith --project my-agent --since 30d
traceport sessions show SESSION_ID --provider langsmith --project my-agent

Each message has a stable index, original character count, omitted count, and SHA-256 hash. Expansion is explicit and applies only to terminal output:

traceport traces show TRACE_ID --provider langsmith --project my-agent --expand 0
traceport traces show TRACE_ID --provider langsmith --project my-agent --expand 0,3
traceport traces show TRACE_ID --provider langsmith --project my-agent --max-chars 5000
traceport traces show TRACE_ID --provider langsmith --project my-agent --full
traceport traces show TRACE_ID --provider langsmith --project my-agent --json

Traceport Agent imports the same traceport.inspection service directly; it does not shell out to the CLI. Before model access, the Agent adds privacy redaction and an independent total-context budget. Context signals use LLM-span token totals rather than copied middleware totals.

Processing loop

The default scan bootstraps seven days, then advances a watermark with a 15-minute overlap and session fingerprints derived from underlying traces and feedback. Unchanged sessions are skipped; sessions that gain traces, change terminal state, or receive new negative feedback are rescanned. It processes at most 500 sessions, uses batches of 20, and allows four concurrent batches.

Each direct Deep Agent returns a required CandidateFindingBatch structured response. Invalid field names, categories, or value types are returned to that agent for correction before host-side Pydantic validation.

Signal coverage

Traceport Agent keeps deterministic checks intentionally small. They only surface explicit telemetry facts; they do not infer behavioral failures. A fact raises priority but does not automatically become an issue. The host passes the screener the exact session/signal pairs that exist. The screener must return exactly one compact SignalAssessment per pair: handled or supports_finding. Missing, duplicate, and extra assessments are rejected. Sessions with no deterministic signals return no assessments, while normal agentic screening still runs.

Everything else is agentic. The screener reads the bounded conversation and compares it with the Agent Overview, configured description, and observed tools. This covers unsupported requests, refusals, looping, context problems, unusual latency or step counts, and other surprising behavior without brittle keyword rules. It can classify the result as feature_gap, missing_capability_awareness, task_evasion, another issue category, or no finding.

External dependency errors are not treated as agent failures when the agent handles them correctly. Provider feedback is read directly by the Agent integration and does not require changing Traceport's public trace models. Feedback comments are redacted and bounded before model access. Scan reports include the number of priority cues and the screener's handled/supports-finding assessment counts.

The evidence thresholds and target-agent context are configurable:

[agent]
description = """
Branna helps winery operators manage production tasks and attached operational evidence.
It should acknowledge user-provided material without claiming actions it did not perform.
"""

[issues]
ordinary_min_evidence_sessions = 2
safety_min_evidence_sessions = 1 # PII leaks and guardrail bypasses
existing_min_evidence_sessions = 1

[evaluators]
min_evidence_traces = 2
min_control_traces = 2
min_sensitivity = 0.8
min_specificity = 0.8

[signals]
negative_feedback_max_score = 0.0
evaluator_failure_max_score = 0.0

The optional description is included in the screener and investigator context, and in the initial Agent Overview. Do not put credentials in it. Defaults preserve the two-session ordinary recurrence rule and the one-session safety exception. Resolved issues reopen on matching evidence; dismissed issues remain suppressed until a human reopens them.

Deep Agent access

The directly invoked agents use a CompositeBackend with ephemeral scratch state and only the read-only routes they need:

No shell backend is installed. The host creates a credential-free HTTPS GitHub checkout, removes write permissions, mounts it only for the scan, and never executes repository code.

Typed tool capabilities are separate from methodology skills. Screeners receive only list_scan_records and get_session_transcript. Only investigators receive get_session_context, and that tool rejects session IDs that were not explicitly flagged. Investigators also receive get_message_chunk for targeted reads from omitted sections of those flagged sessions; screeners do not. The list tool returns metrics and signals without transcript content, preventing duplication when the screener reads each conversation through the transcript tool. Screener messages are bounded to 1,000 characters each and 12,000 characters total by default. Investigator messages are bounded to 4,000 characters each and 30,000 characters total by default; span metadata remains content-free. The investigator can selectively read redacted chunks from omitted sections. Each chunk is capped at the per-message limit, and all chunk reads for a session share the investigator total-character budget. There is no unrestricted full-content tool. Canonical issue, evaluator, memory, and handoff writes remain host-controlled and are not tools on either analysis agent.

The bundled original skills are:

• issue-lifecycle
• trace-triage
• issue-investigation
• memory-curation
• coding-agent-handoff

Add user-managed skill directories with extra_skill_dirs in the TOML configuration. Traceport Agent does not vendor LangChain's eval-engineering skill or Phoenix skills.

extra_skill_dirs = ["/opt/my-agent-skills"]

State and contracts

state/
├── memories/AGENTS.md
├── overview-approval.json
├── cursor.json
├── scans/
├── issues/
├── evaluators/
├── regression-examples/
└── handoffs/

Canonical artifacts are schema-validated JSON with schema_version: 1. Markdown issue and evaluator files are regenerated human projections. Writes use a same-directory temporary file, fsync, and atomic replace. An advisory state lock prevents overlapping runs. Each state directory is bound to one provider/project/repository identity.

Only session IDs, underlying trace IDs, metrics, and capped redacted excerpts are persisted by default. Full sessions and traces remain in memory during the current run.

Investigator bounds are configurable without changing Traceport's lossless source objects:

[privacy]
screener_message_chars = 1000
screener_total_chars = 12000
investigator_message_chars = 4000
investigator_total_chars = 30000
extra_secret_patterns = []

Evaluators

Structural evaluators use a whitelisted predicate DSL over trace status, tool counts, repeated calls, errors, latency, tokens, metadata, contains, and bounded regex. Arbitrary code cannot be represented or executed. Semantic evaluators require a structured Boolean judge verdict plus an explanation.

The host derives a recurrence evaluator from the confirmed issue and tests it as its own pipeline step. It becomes validated_on_sample only when it meets all four thresholds from [evaluators]; otherwise it remains proposed with its failed cases recorded. Skipped cases do not count, and tool or model failures remain infrastructure errors.

Regression assertion creation is a separate step and does not depend on evaluator promotion. Examples retain redacted inputs/context and grounded assertions. Failed production answers are evidence, never ground truth.

Container

Build from this package directory:

docker build -t traceport-agent:0.1.0 .
docker run --rm \
  --env-file traceport-agent.env \
  -v "$PWD/traceport-agent.toml:/config/traceport-agent.toml:ro" \
  -v "$PWD/state:/state" \
  traceport-agent:0.1.0 \
  --config /config/traceport-agent.toml --state /state run

See examples/ for a Kubernetes CronJob and GitHub Actions schedule.

Development

uv sync
uv run ruff format --check src tests
uv run ruff check src tests
uv run mypy src
uv run pytest
uv build

Live provider and model tests are intentionally opt-in and credential-gated.

Traceport Agent depends on the Traceport 0.2.x public API and imports its bounded inspection service directly.

Methodology and attribution

The compact-screening, specialized-investigation, persistent-memory, and discovery-before-fixing workflow is informed by LangSmith Engine's technical write-up and Engine documentation. The tool/skill separation and telemetry-first diagnostic boundary are also informed by Arize's agent-harness article.

Deep Agents is used as an MIT-licensed dependency. The six Traceport Agent skills are original implementations of public methodology. This package does not copy or vendor LangChain's eval-engineering skill or Phoenix skills; users may mount separately obtained skills through extra_skill_dirs.