Phthos Eval

Score an agent run, not a chat reply.

An agent can answer fluently and still call the wrong tool, loop, blow the budget, or break policy. Phthos Eval reads recorded traces (LLM steps + tool calls) and writes a diagnosis JSON you can gate CI on or hand to another system. It does not rewrite prompts, open PRs, or fine-tune.

Python 3.11+. Install from PyPI:

pip install phthos-eval

PyPI · GitHub

flowchart LR
  subgraph stacks [Your agent]
    LC[LangChain]
    ADK[Google ADK]
    OTH[CrewAI / custom]
  end
  stacks --> spans["spans: llm + tool"]
  spans --> scorers[Phthos Eval scorers]
  scorers --> dx[diagnosis.json]
  dx --> ci[CI gate]
  dx --> hint[change_class hint]
  hint -.-> fix[You / other product applies the fix]

What you get

1. Deterministic checks (no API key): expected tools, argument shape, cost/step budget, deny-list policy, repeated tool loops.
2. A success profile, not one score: task_success (pass^N), n_run_reliability (mean repeat pass-rate), pass_at_n (pass@N), plus cost-per-task, p95 latency, tool steps, policy hits.
3. A diagnosis file: scores, typed failures with span ids, and a change_class hint (tool, policy, prompt, …).

Optional: an LLM judge using your key. Without a key, everything above still runs.

Sample scores (bundled fixture)

python -m phthos_eval run -d fixtures/dataset.json — 5 cases, 2 runs each. One case is clean; the others are seeded failures.

Support-agent dogfood (examples/support_agent/dataset.json): task_success 0.50, cost_mean 0.002, latency_p95_ms 230, policy_hits 2, change_class policy. status-ok passes; refund-denied fails.

Any agent stack (LangChain, Google ADK, …)

Phthos Eval does not import LangChain, Google ADK, CrewAI, LlamaIndex, AutoGen, or similar. Those packages run the agent. We score what it did.

Seamless here means one shared trace shape, not a plugin inside each framework:

flowchart TB
  LC[LangChain / LangGraph]
  ADK[Google ADK]
  CR[CrewAI / AutoGen / custom]
  OT[OpenTelemetry / OpenInference]
  LC --> MAP[callbacks / export / your logger]
  ADK --> MAP
  CR --> MAP
  OT --> MAP
  MAP --> SP["spans: id, type llm or tool, name, args, cost"]
  SP --> PE[phthos-eval]
  PE --> DJ[diagnosis.json]

Today: you map your framework’s events into that JSON (a small callback or post-run dump). Then phthos-eval run is the same for every stack.

Typical mappings

You do not wrap the agent in a Phthos runtime. Swap LangChain for ADK and the eval file stays valid as long as spans still look like the table above.

Live: POST /v1/traces with that span JSON, or OTLP/HTTP JSON (openinference.span.kind / gen_ai.tool.name) to a self-hosted engine. See Live engine.

Quick start

Save traces as JSON (see Dataset format), then:

python -m phthos_eval run -d eval/dataset.json -o diagnosis.json
python -m phthos_eval check diagnosis.json

Fail CI when anything is wrong:

python -m phthos_eval run -d eval/dataset.json -o diagnosis.json --fail-on-findings

Try the bundled examples after cloning this repo:

python -m phthos_eval run -d fixtures/dataset.json -o diagnosis.json
python -m phthos_eval run -d examples/support_agent/dataset.json -o diagnosis.json

Live engine (self-host)

Same scorers as offline, on a sampled production stream. You run the process; data stays on your machine. This is not a hosted SaaS and not a LangSmith clone.

sequenceDiagram
  participant Agent
  participant Engine as Live engine
  participant Worker as Async scorers
  Agent->>Engine: POST /v1/traces
  Engine-->>Agent: 202 accepted (sampled or not)
  Note over Agent: Agent request already finished
  Engine->>Worker: if sampled (~5%)
  Worker->>Worker: same scorers as offline
  Worker-->>Engine: diagnosis.json in SQLite

# default sample rate 5%, judge off
phthos-eval live -c examples/live/config.json

# or
docker compose up --build

UI at http://127.0.0.1:8765 — pass rate, cost, policy hits, open a run to see diagnosis JSON. No prompt editor.

Ingest does not wait for scoring:

from phthos_eval.live import LiveClient

client = LiveClient("http://127.0.0.1:8765")
client.ingest(spans=[...], agent_id="support", expected_tools=["search"])

GET /v1/scores · GET /v1/diagnoses/{id} · POST /v1/diagnoses/{id}/export writes an offline dataset you can phthos-eval run.

Demo (forces 100% sample — not for production):

PHTHOS_EVAL_SAMPLE_RATE=1 docker compose up --build
phthos-eval live-demo

Full Compose / OTel / cost knobs: examples/live/README.md.

Do not bankrupt yourself: default is 5% sample and no LLM judge. A judge key on 100% of live traffic is usually more expensive than the agent. Opt in with --live-judge plus OPENAI_API_KEY / PHTHOS_EVAL_API_KEY only if you accept that bill.

Integrate in a project

1. Record traces

When your agent runs (tests or a small harness), write spans like:

{
  "spans": [
    { "id": "s0", "type": "llm", "latency_ms": 120, "cost_usd": 0.002 },
    {
      "id": "s1",
      "type": "tool",
      "name": "lookup_order",
      "args": { "order_id": "A-100" },
      "latency_ms": 30,
      "cost_usd": 0.0
    }
  ]
}

You do not run the agent through Phthos Eval. You export what it did, then score the export.

2. Put cases in a dataset

One file per suite. Each case needs n_runs traces (default in examples: 2) so reliability is real.

{
  "id": "my-agent",
  "n_runs": 2,
  "budget": { "max_cost_usd": 0.05, "max_steps": 8 },
  "policy": { "deny_tools": ["issue_refund"] },
  "tool_schemas": {
    "lookup_order": { "required": ["order_id"] }
  },
  "cases": [
    {
      "id": "status-ok",
      "expected_tools": ["lookup_order"],
      "traces": [{ "spans": [] }, { "spans": [] }]
    }
  ]
}

3. Call from Python (pytest)

import json
from pathlib import Path
from phthos_eval import run_dataset, validate_diagnosis

def test_agent_eval():
    dataset = json.loads(Path("eval/dataset.json").read_text())
    doc = run_dataset(dataset)
    assert validate_diagnosis(doc) == []
    assert doc["change_class"] == "none"
    assert doc["scores"]["n_run_reliability"] == 1.0

Custom check (still deterministic — no LLM):

from phthos_eval import failure, run_dataset

class NoEmptyTrace:
    def score(self, trace, *, case, dataset, case_id, trace_index):
        if not trace.get("spans"):
            return [failure("policy", "empty", case_id=case_id, trace_index=trace_index)]
        return []

doc = run_dataset(dataset, scorers=[NoEmptyTrace()])  # replaces defaults; add default_scorers() to keep them

To keep built-in scorers and yours:

from phthos_eval import default_scorers, run_dataset

doc = run_dataset(dataset, scorers=[*default_scorers(), NoEmptyTrace()])

4. GitHub Actions

- uses: actions/setup-python@v5
  with:
    python-version: "3.12"
- run: pip install phthos-eval
- run: python -m phthos_eval run -d eval/dataset.json -o diagnosis.json --fail-on-findings

Full copy-paste: examples/github-eval.yml.

Dataset format

Tool spans: name, args, optional cost_usd, latency_ms.

Metrics (what they mean)

Suite-level diagnosis.json → scores. This is an AgentSLABench-style profile (success under cost/latency/policy), not a single 0–1 judge number.

Success (codegen / agent standard: pass^k vs pass@k):

On the bundled fixture every case is all-pass or all-fail, so the three numbers match. They diverge as soon as a case is mixed (see tests).

Cost, latency, steps (ops / SLA):

Safety / typed hits:

judge.score (0–1) appears only if you set a judge key. Treat it as extra signal, not the verdict. We do not ship hallucination/fluency vanity scores — those are judge-only and not the contract.

Per case: cases[] has passed, pass_rate, cost, latency_ms, steps, and that case’s failures.

Failures and change_class

Each failure has a type, a span_id, and evidence (span / step / case / which of the N traces). Types:

change_class is a hint for whatever improves the agent (you, CI, a later tool). This package does not apply the change.

Values: prompt · tool · policy · model · finetune_data · none (clean run).

Optional LLM judge

Not required. Deterministic scorers always run.

Do not reuse the agent’s production keys as the judge unless you intend that. Agent keys run the system under test; judge keys only score. With no judge key, judge.skipped is true and reason is no_key.

What this is not

• Not LangSmith (no prompt playground, no hosted cloud accounts).
• Not an auto-fixer or fine-tuner. Export a failing live run; you (or another product) apply the change.
• Not a hosted LLM. OSS / self-host uses your machine and, optionally, your judge key.

Develop this repo

pip install -e ".[dev]"
python -m pytest
python -m ruff check src tests

License: MIT.