costtrace 0.1.0

A deterministic cost-attribution profiler for LLM traces — an X-ray for where your token spend goes.

CostTrace

An X-ray for your LLM token spend.

CostTrace is a deterministic cost-attribution profiler for LLM applications. You add one line after each agent run to capture the durable identity of your billed calls. Later, CostTrace re-fetches the exact usage from the provider, prices it, and tells you three things most dashboards can't:

1. Where your money goes — split across system prompt, conversation history, retrieved context, the user's turn, and the model's output.
2. What changed over time — cost-per-turn jumps, model switches, history bloat, longer outputs, cache regressions, or system-prompt edits.
3. Why — an LLM-as-judge explains the most likely driver in plain English, grounded strictly in your real numbers.

Integrations: today CostTrace supports the OpenAI Agents SDK (Python) only. The design is provider-tagged, and support for more SDKs and providers is planned (see Roadmap).

---

Table of contents

• Why CostTrace
• How it's different
• Install
• Quickstart
• Core concepts
• Providers
• CLI reference
• Programmatic API
• How it works
• The cost model
• Data, privacy & retention
• Configuration
• Development
• Roadmap
• FAQ
• License

---

Why CostTrace

LLM bills are opaque. A provider hands you a token count per call, not dollars, and never tells you which part of the prompt cost the money. In a multi-turn agent, the conversation history is re-sent on every turn, a verbose system prompt is billed on every call, and a single model swap can 100× your cost — but the invoice just says "tokens."

CostTrace turns that opaque number into an itemized, timestamped, queryable picture of your spend, so you can answer questions like:

• "What fraction of my bill is just re-sending history?"
• "Our costs doubled last week — what changed?"
• "How much cheaper is nano than pro for the same workload?"
• "Did that system-prompt edit break prompt caching?"

How it's different

• Capture is tiny and cheap. We store only {provider, conversation_id, response_ids} — no usage numbers, no message content. Everything else is re-fetched on demand (provider GET reads are not billed).
• Accounting is deterministic. Costs are computed by us as rate × tokens using live LiteLLM pricing (with a bundled offline fallback) — reproducible, auditable, no guessing.
• Detection is general, not single-cause. Rather than blaming one thing, CostTrace compares a recent window against a prior one across every dimension and surfaces each signal that moved.
• The LLM only explains, never computes. The judge receives a compact, number-grounded digest and is instructed to reason over it — it never invents figures.
• It's a profiler, not a proxy. No gateway to route traffic through, no latency added to your requests. You capture ids; analysis happens offline, on demand.

Install

pip install costtrace

This installs the importable CostTrace package and the costtrace CLI.

Set your OpenAI key via the environment or a .env in your project root:

export OPENAI_API_KEY=sk-...
# or, in ./.env
# OPENAI_API_KEY=sk-...

Requires Python 3.9+.

Quickstart

1. Capture — one line after each agent run

from agents import Agent, OpenAIConversationsSession, Runner
from CostTrace import capture

session = OpenAIConversationsSession()
result = await Runner.run(agent, user_input, session=session)

capture(result, conversation_id, provider="openai_agents")   # -> ./.costtrace/captures.json

capture() extracts the response ids from the run, validates them, de-dupes, timestamps them, and appends them under the conversation id. It's idempotent and prunes anything older than the retention window on every write.

provider is required — it records which backend the ids came from so they can be re-fetched later. Passing an unknown/missing provider raises an error that lists the available providers.

You can also pass a plain list of ids instead of a RunResult:

capture(["resp_abc", "resp_def"], conversation_id, provider="openai_agents")

2. Analyze — from the CLI

costtrace analyze                              # everything captured
costtrace analyze --limit 5                    # 5 most recent conversations
costtrace analyze --date 2026/07/20            # a single day
costtrace analyze --starting-date 2026/07/01   # from that day until today
costtrace analyze --conversationId conv_abc    # one conversation
costtrace analyze --no-judge                   # skip the LLM explanation

Example output:

======================================================================
WHERE THE MONEY GOES
======================================================================
sessions: 3   total: $6.143908   avg cache-hit: 53%
  output      85.9%   $5.275134
  history     14.0%   $0.857593
  system       0.1%   $0.009336
  user         0.0%   $0.001846
  retrieved    0.0%   $0.000000

======================================================================
PER-MODEL COMPARISON
======================================================================
  model               cost/turn   cache  out tok/turn  turns
  gpt-5.4-pro        $ 1.118778     0%          5376      5
  gpt-5.4            $ 0.043458    80%          2305     12
  gpt-5.4-nano       $ 0.002377    78%          1472     12

  gpt-5.4-pro costs 470.6x more per turn than gpt-5.4-nano

======================================================================
WHAT CHANGED (deterministic signals)
======================================================================
  [1] (medium) dominant_bucket: 'output' is 86% of total spend ($5.275134)

======================================================================
JUDGE (LLM explanation)
======================================================================
Output dominates spend; the cost gap tracks the switch to gpt-5.4-pro
(higher rate + 3.6x longer outputs + 0% cache vs ~80%). Consider routing
routine turns to gpt-5.4-nano and capping output length.

Core concepts

Term	Meaning
capture	The dev-side one-liner. Persists {provider, conversation_id, response_ids} — the durable identity of billed calls, nothing else.
conversation	One session, grouped by the provider's conversation id.
call	One billed model request == one response_id.
message kind	Each item in a call is classified as system, history, retrieved, user, or output.
bucket	Cost aggregated by message kind — the "where the money goes" split.
session metric	One durable, timestamped row per conversation (cost, buckets, cache rate, cost/turn, model, system-prompt fingerprint).
provider tag	Routes fetching to the right backend. Currently only openai_agents.

Providers

provider is a required argument to capture() — it records which backend the response ids belong to, so CostTrace knows where to re-fetch usage from.

Provider tag	Backend	Status
openai_agents	OpenAI Agents SDK (Python)	✅ Available
(more)	OpenAI Python SDK, Anthropic, Langfuse/Helicone exports, …	🚧 Planned (roadmap)

Passing an unknown or missing provider fails fast with an error that lists the currently supported tags:

>>> capture(result, conversation_id)                     # missing
CaptureSchemaError: provider is required — pass one of the supported providers:
['openai_agents'] (e.g. capture(result, conversation_id, provider='openai_agents'))

>>> capture(result, conversation_id, provider="anthropic")   # unknown
CaptureSchemaError: unknown provider 'anthropic'; supported providers: ['openai_agents']

The available tags are also exposed programmatically:

from CostTrace import Provider
Provider.values()        # ['openai_agents']

CLI reference

costtrace analyze [options]

Option	Description
--limit N	Analyze the N most recent conversations.
--conversationId conv_xxx	Analyze a single conversation (alias: --conversation-id).
--date yyyy/mm/dd	Only conversations from that day.
--starting-date yyyy/mm/dd	Conversations from that day until today.
--no-judge	Skip the LLM explanation (fully offline).

--date and --starting-date are mutually exclusive. Run costtrace --help or costtrace analyze --help for the full, self-documenting listing.

Programmatic API

from CostTrace import load_captures, fetch_captures, Pricer

captures = load_captures(".costtrace/captures.json")   # validated ids
report = fetch_captures(captures)                       # validated traces

report.conversations   # list[ConversationTrace] — ready to analyze
report.rejections      # ids dropped (expired/invalid), each with a reason
report.fetched_calls   # count of successfully fetched calls

Build your own analysis with the analysis module:

from CostTrace.analysis import (
    compute_session_metric, upsert_metrics, load_metrics,
    summarize_overall, summarize_by_model, detect_all, judge,
)

pricer = Pricer.default()  # live LiteLLM pricing, cached, yaml fallback
metrics = [compute_session_metric(c, pricer) for c in report.conversations]
upsert_metrics(metrics)                       # persist to .costtrace/metrics.jsonl

rows = load_metrics()
overall = summarize_overall(rows)
findings = detect_all(overall, rows)
verdict = judge(overall, findings, by_model=summarize_by_model(rows))
print(verdict.verdict)

How it works

                your app
                   │  capture(result, conversation_id)
                   ▼
   .costtrace/captures.json   ── just ids, 15-day retention, auto-pruned
                   │
   costtrace analyze:
                   ▼
   ingest ──► fetch ──► metrics ──► store ──► aggregate ──► detect ──► judge
   (validate  (re-fetch  (bucket    (durable   (overall +   (what      (LLM
    ids)       usage &    split +    JSONL)     per-model +  changed)    explains
               content)   cost)                 per-prompt)             why)

1. Ingest — strict schema + id-format validation; clear errors; duplicate conversations merged.
2. Fetch — for each id, pull the response (model + usage) and its input items from the provider, then classify each message (system/history/ retrieved/user/output). Resilient: expired or invalid ids are recorded as rejections and skipped; only auth failures are fatal.
3. Metrics — per conversation, compute total cost and split the input cost across buckets by token share (estimated with a tokenizer, then scaled so the parts sum to the provider's real input_tokens). Output cost is exact.
4. Store — append/upsert one row per conversation to .costtrace/metrics.jsonl; refresh rows that gained turns; prune expired rows. This history survives the id-retention window.
5. Aggregate — overall spend split, per-model comparison, per-prompt-version grouping.
6. Detect — compare a recent window vs a prior one and flag every dimension that moved: cost_increase, model_shift, bucket_shift, cache_drop, output_growth, prompt_change, plus a static dominant_bucket.
7. Judge — assemble a compact, grounded digest and ask an LLM to name the likely cause and a fix. It reasons over the signals; it does not compute numbers.

The cost model

Providers return tokens, not dollars — and never a per-call price. CostTrace computes cost itself:

cost = uncached_input_tokens × input_rate
     + cached_input_tokens   × cache_read_rate
     + output_tokens         × output_rate

Rates come from LiteLLM's community-maintained pricing table (≈3,000 models), fetched and cached locally, with a small bundled pricing.yaml as an offline fallback. Because accounting is pure arithmetic on real token counts, results are reproducible and auditable.

The bucket split for the input side is an approximation: the provider bills one input_tokens number per call, so CostTrace estimates each message's share with a tokenizer and scales those shares to sum exactly to the real input cost. The output bucket is exact.

Data, privacy & retention

• What's stored on disk: only response/conversation ids in .costtrace/captures.json, plus computed metrics (numbers, model names, and a hash + copy of the system prompt) in .costtrace/metrics.jsonl, plus a cached pricing table. No user/assistant message content is persisted.
• Where: a project-local .costtrace/ directory, discovered by walking up to the nearest .git / pyproject.toml (like git finds its root), so capture() and the CLI always agree regardless of your working directory. Override with COSTTRACE_HOME.
• Retention: ids are only re-fetchable from the provider for a limited window, so captures older than 15 days are pruned automatically on write — you lose nothing that could still be fetched.
• The judge digest contains aggregate numbers and (only when a prompt change is detected) the system-prompt text — never raw conversation content.
• Secrets: your API key is read from the environment or a .env; it is never written to the data directory.

Configuration

Variable	Purpose
OPENAI_API_KEY	Required for fetching usage and for the LLM judge.
COSTTRACE_HOME	Override the .costtrace/ data directory location.

Development

git clone https://github.com/sanat77/CostTrace.git
cd CostTrace
python -m venv .venv && source .venv/bin/activate
pip install -e .[dev]

python -m pytest -q                       # run the test suite
python -m pyflakes src tests conftest.py  # lint

The package source lives in src/ and is mapped to the import name CostTrace via package-dir in pyproject.toml, so the built wheel ships CostTrace/ and pip install / import CostTrace are unaffected by the on-disk layout. A conftest.py replicates that mapping so tests import from the source tree without installing.

Roadmap

CostTrace v1 targets the OpenAI Agents SDK (Python) (provider = "openai_agents"). Planned next:

• More Python LLM SDKs / providers (e.g. the OpenAI Python SDK directly, Anthropic, and others) behind the same provider-tagged architecture.
• Ingesting exports from existing observability tools (e.g. Langfuse / Helicone) as additional provider tags.
• Longer retention & trend history as an opt-in tier.
• Richer detectors and per-tenant / per-feature attribution.

Contributions and provider requests are welcome — open an issue.

FAQ

Does this add latency to my app? No. You only save ids at runtime; all fetching and analysis happens later, offline.

Do the re-fetches cost money? No — provider GET reads of past responses are not billed. Only your original generations were.

Why can't I analyze conversations from last month? Ids expire at the provider (~retention window), so they can't be re-fetched. The computed metrics you already stored remain, but new deep analysis needs live ids.

Distribution name vs import name? You pip install costtrace (lowercase) and import CostTrace (capitalized) — standard PyPI convention.

License

MIT — see LICENSE.