tracepress 0.0.3

Find failures, PII, frustration, or misbehavior in agent traces.

Run it in the browser on eight failed runs: Colab demo.

pip install tracepress

import tracepress as tp
tp.configure(lm=tp.Model("openrouter:openai/gpt-6-luna", temperature=1, max_tokens=800, reasoning="off"))

ws = tp.Workspace()

@ws.pipeline(description="Summarizes each step of a failed run, then explains why the run failed.")
def failure_reasons(steps):
    summarized = steps.lm_map(
        "task_name, step_id, source, step -> step_summary",
        "Summarize what happened in this step in one sentence.",
        save_as="step_summaries",
    )
    return summarized.lm_agg(
        "task_name, failed_tests, step_summary -> failure_reason",
        "Explain in one sentence why the run failed.",
        by="session_id",
        save_as="failure_reasons",
    )

reasons = failure_reasons(steps)  # one row per step in, one row per run out

steps is one row per step of a failed run: the task, the step text, and the failed tests. reasons is one row per run:

Use it from Claude Code

TracePress ships skills that teach a coding agent (Claude Code, Codex, Cursor, and others) to build pipelines and dig through their artifacts:

npx skills add Ziems/tracepress

Then ask in plain words, for example: "Load the Harbor job in jobs/my-job, find out why the failed runs failed, and write a failure-mode report with counts."

• tracepress-pipelines writes a pipeline file in your project, tries it on a few rows, and runs it under a budget.
• tracepress-explore answers questions from a workspace's artifacts with pandas, runs extra operators on subsets only when it has to, and cites artifact names and row ids.

The skills live in skills/.

examples/tracepress_gpt6_luna.ipynb is the same analysis on 30 real runs in examples/data. examples/quickstart.ipynb is a shorter walkthrough, and REFERENCE.md says what each piece is for.

Operators

Operators are DataFrame methods starting with lm_. Each takes a signature that says which columns go in and which come out, and an instruction in plain prose. Signatures are DSPy signatures, so the two stay apart. Each returns a DataFrame. None is terminal, so any chain works, including an aggregate of an aggregate.

# one call per row; the outputs become columns. Type them (bool, int, float, list[str], Literal[...]) or pass examples= for few-shot rows.
steps.lm_map("task_name, step_id, source, step -> step_summary", "Summarize what happened in this step in one sentence.")

# keep the rows where the condition holds. The index is preserved.
steps.lm_filter("step", "The model edits a file or runs a test")

# a hierarchical reduce to one row per by group, for any number of rows.
summaries.lm_agg("task_name, failed_tests, step_summary -> failure_reason", "Why did the run fail?", by="session_id")

# listwise ranking in rounds. Adds a rank column.
steps.lm_topk("step", "Most likely the step where the run went wrong", k=3)

# LLM-only duplicate clustering. Keeps one row per cluster.
reasons.lm_dedup("task_name, failure_reason", "Same failure for the same reason")

lm_filter, lm_topk and lm_dedup only read rows, so their signature is just the inputs. DSPy doesn't allow an input and an output with the same name, so "v -> v" won't work. When fields need descriptions, write the signature as a class; its docstring is the instruction. tp.Signature, In and Out are DSPy's Signature, InputField and OutputField:

class SummarizeStep(tp.Signature):
    """What happened in this step of a failed agent run?"""
    task_name: str = tp.In()
    step_id: int = tp.In()
    source: str = tp.In()
    step: str = tp.In(desc="the step's message, tool calls, and observation")
    step_summary: str = tp.Out(desc="one sentence")

steps.lm_map(SummarizeStep)

Inside a pipeline, save_as="name" names the artifact.

Models

tracepress.Model(...) wraps a dspy.LM running on the lm15 engine. A direct model is a LiteLLM id: openai/gpt-5-nano, anthropic/claude-opus-5-5. A gateway prefixes that id: openrouter:openai/gpt-6-luna, portkey:openai/gpt-5-nano. DSPy supplies prompt formatting (its XML adapter), parsing, retries, the response cache (~/.tracepress/cache), and per-call cost. TracePress adds:

• concurrent batches (max_workers) with a progress bar
• token and cost totals on lm.usage and in every artifact's manifest
• reasoning="off" | "low" | ..., spelled correctly for each provider
• engine= for a custom lm15 engine (the tests use this); any other keyword goes to dspy.LM

with tracepress.budget(max_calls=..., max_usd=...) refuses a batch before it starts if it would go over the cap. For dollar caps, pass price=(input, output) in $/M tokens to get an estimate up front; without it, the cap is checked against what has actually been spent.

Portkey

Set PORTKEY_API_KEY. The string after portkey: is a LiteLLM id, and the call sends that id with a leading @ to Portkey:

tp.configure(lm=tp.Model("portkey:openai/gpt-5-nano"))

portkey:@openai/gpt-5-nano is the same call. A gateway uses its own LiteLLM rate when that entry has one, and otherwise the rate for the id after the colon, so openrouter:openai/gpt-6-luna takes the OpenAI price. A catalog slug LiteLLM does not know warns and records $0 until you pass a rate, in dollars per million tokens:

tp.Model("portkey:@openai-prod/gpt-5-nano", price=(0.05, 0.40))

Workspace

Each ws.run (or call to a registered pipeline) writes workspace/runs/<run_id>/, which holds a manifest.json and one parquet file per artifact. Every df.lm_* call inside a run is saved automatically. The artifact's name comes from save_as=, or is step{n}_{op} when none is given. The manifest records the op, the instruction, the model, the parent artifacts, the shape, and the usage. A run that fails keeps the artifacts it finished.

• ws.artifacts() lists artifacts.
• ws["name"] loads the latest artifact with that name; ws["run_id/name"] loads a specific one.
• ws.lineage(name) traces an artifact back to its inputs.
• ws.save(name, df) adds raw data.

Pipelines are code, so re-run the cell that registers them in each new session. Their artifacts stay on disk.

Harbor

tp.harbor_load reads a Harbor job directory, a trial directory, or an ATIF trajectory.json into one row per trial and one row per step. The classifiers take that already-loaded result (or a steps DataFrame from it), not a path.

loaded = tp.harbor_load("jobs/my-job")
labels = tp.harbor_agent_taxonomy(loaded)      # summarize each step, then the nine agent trajectory taxonomy modes per trial
commands = tp.harbor_command_taxonomy(loaded)  # one row per failed command

@ws.pipeline
def harbor_agent_taxonomy(steps):
    return tp.harbor_agent_taxonomy(steps)

@ws.pipeline
def harbor_command_taxonomy(steps):
    return tp.harbor_command_taxonomy(steps)

labels.modes lists every mode the trial shows. commands.subcategory is one command-failure subcategory. Pass lm= to use a model other than the configured one. harbor_load is not a pipeline: call it yourself and pass the result into a pipeline.

Credit

TracePress is a fork of LOTUS (LLMs Over Text, Unstructured and Structured data), from Liana Patel at UC Berkeley.