Phthos Eval

Score an agent run, not a chat reply.

An agent can answer fluently and still call the wrong tool, loop, blow the budget, or break policy. Phthos Eval reads recorded traces (LLM steps + tool calls) and writes a diagnosis JSON you can gate CI on or hand to another system. It does not rewrite prompts, open PRs, or fine-tune.

Python 3.11+. Install from PyPI:

pip install phthos-eval

PyPI · GitHub

flowchart LR
  subgraph stacks [Your agent]
    LC[LangChain]
    ADK[Google ADK]
    OTH[CrewAI / custom]
  end
  stacks --> spans["spans: llm + tool"]
  spans --> scorers[Phthos Eval scorers]
  scorers --> dx[diagnosis.json]
  dx --> ci[CI gate]
  dx --> hint[change_class hint]
  hint -.-> fix[You / other product applies the fix]

What you get

1. Deterministic checks (no API key): expected tools, argument shape, cost/step budget, deny-list policy, repeated tool loops.
2. A success profile, not one score: task_success (pass^N), n_run_reliability (mean repeat pass-rate), pass_at_n (pass@N), plus cost-per-task, p95 latency, tool steps, policy hits.
3. A diagnosis file: scores, typed failures with span ids, and a change_class hint (tool, policy, prompt, …).

Optional: an LLM judge using your key. Without a key, everything above still runs.

Sample scores (bundled fixture)

python -m phthos_eval run -d fixtures/dataset.json — 5 cases, 2 runs each. One case is clean; the others are seeded failures.

Support-agent dogfood (examples/support_agent/dataset.json): task_success 0.50, cost_mean 0.002, latency_p95_ms 230, policy_hits 2, change_class policy. status-ok passes; refund-denied fails.

Any agent stack (LangChain, Google ADK, …)

Phthos Eval does not import LangChain, Google ADK, CrewAI, LlamaIndex, AutoGen, or similar. Those packages run the agent. We score what it did.

Seamless here means one shared trace shape, not a plugin inside each framework:

flowchart TB
  LC[LangChain / LangGraph]
  ADK[Google ADK]
  CR[CrewAI / AutoGen / custom]
  OT[OpenTelemetry / OpenInference]
  LC --> MAP[TraceSink.wrap / OTel]
  ADK --> MAP
  CR --> MAP
  OT --> MAP
  MAP --> SP["spans: id, type llm or tool, name, args, cost"]
  SP --> PE[phthos-eval]
  PE --> DJ[diagnosis.json]

Today: wrap the agent you already have. TraceSink attaches collectors; the framework still runs the agent. We only score spans.

from phthos_eval import TraceSink

sink = TraceSink()
agent = sink.wrap(agent)   # see TraceSink.frameworks
# run the agent as usual, then:
doc = sink.diagnose(expected_tools=["search"])

Unknown stack: sink.add_llm(...) / sink.add_tool(...), or export OpenTelemetry OpenInference to POST /v1/otel/traces. Examples: agent_integration_examples/.

You do not replace the agent runtime. Swap LangChain for ADK and the eval file stays valid as long as spans are still llm / tool JSON.

Live: POST /v1/traces with that span JSON, or OTLP/HTTP JSON (openinference.span.kind / gen_ai.tool.name) to a self-hosted engine. See Live engine.

Quick start

Save traces as JSON (see Dataset format), then:

python -m phthos_eval run -d eval/dataset.json -o diagnosis.json
python -m phthos_eval check diagnosis.json

Fail CI when anything is wrong:

python -m phthos_eval run -d eval/dataset.json -o diagnosis.json --fail-on-findings

Try the bundled examples after cloning this repo:

python -m phthos_eval run -d fixtures/dataset.json -o diagnosis.json
python -m phthos_eval run -d examples/support_agent/dataset.json -o diagnosis.json
# Wrap + score: python agent_integration_examples/google_adk/lib/agent.py
# Wrap + live: python agent_integration_examples/google_adk/live/agent.py

Live engine (self-host)

Same scorers as offline, on a sampled production stream. You run the process; data stays on your machine. This is not LangSmith. Optional hosted mode (--hosted) is the same binary with login and tenants — OSS self-host stays the default.

sequenceDiagram
  participant Agent
  participant Engine as Live engine
  participant Worker as Async scorers
  Agent->>Engine: POST /v1/traces
  Engine-->>Agent: 202 accepted (sampled or not)
  Note over Agent: Agent request already finished
  Engine->>Worker: if sampled (~5%)
  Worker->>Worker: same scorers as offline
  Worker-->>Engine: diagnosis.json in SQLite

# default sample rate 5%, judge off
phthos-eval live -c examples/live/config.json

# or
docker compose up --build

UI at http://127.0.0.1:8765 — pass rate, cost, policy hits, open a run to see diagnosis JSON. No prompt editor.

Ingest does not wait for scoring:

from phthos_eval.live import LiveClient

client = LiveClient("http://127.0.0.1:8765")
client.ingest(spans=[...], agent_id="support", expected_tools=["search"])

GET /v1/scores · GET /v1/diagnoses · GET /v1/diagnoses/{id} · POST /v1/compare · POST /v1/diagnoses/{id}/export writes an offline dataset you can phthos-eval run. After you change the agent: phthos-eval compare --before a.json --after b.json. Contract: docs/CONSUMER.md. Example: examples/consumer/.

Demo (forces 100% sample — not for production):

PHTHOS_EVAL_SAMPLE_RATE=1 docker compose up --build
phthos-eval live-demo

Full Compose / OTel / cost knobs: examples/live/README.md.

Do not bankrupt yourself: default is 5% sample and no LLM judge. A judge key on 100% of live traffic is usually more expensive than the agent. Opt in with --live-judge plus OPENAI_API_KEY / PHTHOS_EVAL_API_KEY only if you accept that bill.

Hosted mode (same engine)

We operate this in cloud; you can also run it yourself. It is not a second product: PHTHOS_EVAL_HOSTED=1 or phthos-eval live --hosted.

• Sign-up / login, isolated tenants, dashboard (live, history, datasets), alerts when pass rate drops
• Default BYOK — traces are not sent to a model we own; live judge still off unless --live-judge and your key
• Export diagnoses + datasets (GET /v1/export) — no hostage data
• Self-host path above stays complete without accounts

phthos-eval live --hosted --host 0.0.0.0
# or
docker compose -f docker-compose.hosted.yml up --build

from phthos_eval.live import LiveClient

client = LiveClient("https://your-eval-url", api_key="pk_…")
client.ingest(spans=[...], expected_tools=["search"])

CI can keep using phthos-eval run locally, or put_dataset / run_dataset on the hosted URL. Compare two runs: POST /v1/compare. Poll: GET /v1/diagnoses?since=. Details: docs/hosted.md, docs/CONSUMER.md. What is stored: docs/PRIVACY.md. GET /status for health.

Paid cloud extras (retention, SAML, hosted judges we meter, seats) are ops, not a paywall on scores. Catalog: docs/PLANS.md. Stripe / SAML UI is the private overlay, not this package.

Integrate in a project

1. Record traces

When your agent runs (tests or a small harness), write spans like:

{
  "spans": [
    { "id": "s0", "type": "llm", "latency_ms": 120, "cost_usd": 0.002 },
    {
      "id": "s1",
      "type": "tool",
      "name": "lookup_order",
      "args": { "order_id": "A-100" },
      "latency_ms": 30,
      "cost_usd": 0.0
    }
  ]
}

You do not run the agent through Phthos Eval. You export what it did, then score the export.

2. Put cases in a dataset

One file per suite. Each case needs n_runs traces (default in examples: 2) so reliability is real.

{
  "id": "my-agent",
  "n_runs": 2,
  "budget": { "max_cost_usd": 0.05, "max_steps": 8 },
  "policy": { "deny_tools": ["issue_refund"] },
  "tool_schemas": {
    "lookup_order": { "required": ["order_id"] }
  },
  "cases": [
    {
      "id": "status-ok",
      "expected_tools": ["lookup_order"],
      "traces": [{ "spans": [] }, { "spans": [] }]
    }
  ]
}

3. Call from Python (pytest)

import json
from pathlib import Path
from phthos_eval import run_dataset, validate_diagnosis

def test_agent_eval():
    dataset = json.loads(Path("eval/dataset.json").read_text())
    doc = run_dataset(dataset)
    assert validate_diagnosis(doc) == []
    assert doc["change_class"] == "none"
    assert doc["scores"]["n_run_reliability"] == 1.0

Custom check (still deterministic — no LLM):

from phthos_eval import failure, run_dataset

class NoEmptyTrace:
    def score(self, trace, *, case, dataset, case_id, trace_index):
        if not trace.get("spans"):
            return [failure("policy", "empty", case_id=case_id, trace_index=trace_index)]
        return []

doc = run_dataset(dataset, scorers=[NoEmptyTrace()])  # replaces defaults; add default_scorers() to keep them

To keep built-in scorers and yours:

from phthos_eval import default_scorers, run_dataset

doc = run_dataset(dataset, scorers=[*default_scorers(), NoEmptyTrace()])

4. GitHub Actions

- uses: actions/setup-python@v5
  with:
    python-version: "3.12"
- run: pip install phthos-eval
- run: python -m phthos_eval run -d eval/dataset.json -o diagnosis.json --fail-on-findings

Full copy-paste: examples/github-eval.yml.

Dataset format

Tool spans: name, args, optional cost_usd, latency_ms.

Metrics (what they mean)

Suite-level diagnosis.json → scores. This is an AgentSLABench-style profile (success under cost/latency/policy), not a single 0–1 judge number.

Success (codegen / agent standard: pass^k vs pass@k):

On the bundled fixture every case is all-pass or all-fail, so the three numbers match. They diverge as soon as a case is mixed (see tests).

Cost, latency, steps (ops / SLA):

Safety / typed hits:

judge.score (0–1) appears only if you set a judge key. Treat it as extra signal, not the verdict. We do not ship hallucination/fluency vanity scores — those are judge-only and not the contract.

Per case: cases[] has passed, pass_rate, cost, latency_ms, steps, and that case’s failures.

Failures and change_class

Each failure has a type, a span_id, and evidence (span / step / case / which of the N traces). Types:

change_class is a hint for whatever improves the agent (you, CI, a later tool). This package does not apply the change. Another system can poll or take a webhook, ship a change, and we re-eval — docs/CONSUMER.md.

Values: prompt · tool · policy · model · finetune_data · none (clean run).

Optional LLM judge

Not required. Deterministic scorers always run.

Do not reuse the agent’s production keys as the judge unless you intend that. Agent keys run the system under test; judge keys only score. With no judge key, judge.skipped is true and reason is no_key.

What this is not

• Not LangSmith (no prompt playground). Hosted mode is login + scores, not a prompt IDE.
• Not an auto-fixer or fine-tuner. Export a failing live run; you (or another product) apply the change. export-finetune is labeled traces for their trainer.
• Not a hosted LLM. Judge is BYOK and off by default on live. Traces are not required to go to a model we own.

Develop this repo

pip install -e ".[dev]"
python -m pytest
python -m ruff check src tests

License: MIT.