trajeval 0.2.0

Grade agent trajectories: tool selection, argument correctness, and end state.

trajeval

Grade agent trajectories on what actually matters: did the agent pick the right tools, call them with correct arguments, and leave the world in the right state?

Zero dependencies. Pure Python ≥3.9. Built to score SRE agents healing broken infrastructure, general enough to grade any tool-using agent.

pip install trajeval

Why

Most agent evals check the final answer. But an agent that "fixed" an outage by restarting everything five times and deleting a volume along the way is not the same as one that read the logs, restarted the one dead dependency, and verified health. Trajectory grading makes that difference measurable — and comparable across models.

Every check belongs to a category, so a leaderboard doesn't just rank models, it tells you how each one fails:

category	question it answers
tool_selection	Did it reach for the right tools, in a sane order, and avoid destructive ones?
arguments	Were the calls made with correct arguments (root cause, not symptom)?
end_state	Is the environment actually fixed, per post-run probes?
efficiency	Did it get there without flailing, repeating itself, or erroring?
output	Did the final diagnosis say the right thing?

Quick start

from trajeval import (
    Rubric, Trajectory, Step, Leaderboard,
    ToolUsed, ToolNotUsed, ToolOrder, ArgMatch,
    EndStateEquals, StepBudget, FinalAnswerMatches,
)

rubric = Rubric(checks=[
    ToolUsed(tool="get_logs"),                                    # diagnose...
    ToolOrder(sequence=["get_logs", "restart_service"], weight=2),  # ...before acting
    ToolNotUsed(tool="delete_volume", weight=2),                  # never destroy data
    ArgMatch(tool="restart_service", expected={"name": "redis"}), # fix the root cause
    EndStateEquals(path="services.payments.healthy", value=True, weight=3),
    StepBudget(budget=6),
    FinalAnswerMatches(pattern="redis"),
])

score = rubric.grade(trajectory)   # Score in [0, 1] + per-check breakdown
print(score.to_markdown())

board = Leaderboard().add(score, *other_scores)
print(board.to_markdown())         # ranked table with category columns

Getting trajectories

From provider transcripts — the adapters pair tool calls with their results by id:

from trajeval import from_anthropic, from_openai

t = from_anthropic(messages, scenario="dead-dependency", model="claude-x",
                   final_state=probe_environment())  # your post-run health probes
t = from_openai(messages, scenario="dead-dependency", model="gpt-x",
                final_state=probe_environment())

Or build them directly / persist them:

from trajeval import Trajectory, Step, save_jsonl, load_jsonl

t = Trajectory(scenario="filled-disk", model="m",
               steps=[Step("get_disk_usage", {"host": "web-1"}, result="97%")],
               final_state={"disk_pct": 41})
save_jsonl([t], "runs.jsonl")

final_state is a plain dict your harness captures after the run (health checks, disk usage, config hashes). End-state checks assert against it with dotted paths — the agent is graded on the world, not on its own claims.

Built-in checks

Tool selection — ToolUsed, ToolNotUsed, ToolOrder (subsequence), FirstTool · Arguments — ArgMatch (subset or exact, nested), ArgPredicate, AllCallsValid (partial credit) · End state — EndStateEquals (dotted path), EndStatePredicate · Efficiency — StepBudget (linear decay past budget), NoRepeatedCalls, NoErrors · Output — FinalAnswerMatches (regex)

Every check accepts gating=False to mark it advisory: it still counts toward the weighted score, but failing it doesn't block Score.passed_all ("solved"). Use it for efficiency checks — a slow fix is still a fix.

Custom checks are one dataclass:

from dataclasses import dataclass
from trajeval import Check
from trajeval.checks import END_STATE

@dataclass
class DiskBelow(Check):
    pct: int = 80
    category: str = END_STATE
    def __post_init__(self): self.name = self.name or f"disk<{self.pct}%"
    def evaluate(self, t):
        return self._result(1.0 if t.final_state.get("disk_pct", 100) < self.pct else 0.0)

Worked example

examples/sre_incident.py grades three simulated agents on a dead-dependency incident — one that diagnoses properly, one that chases symptoms, one that flails and reaches for delete_volume:

| rank | model | overall | solved | tool_selection | arguments | end_state | efficiency | output |
|---|---|---|---|---|---|---|---|---|
| 1 | frontier-model | 1.00 | 1/1 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 2 | mid-model | 0.99 | 0/1 | 1.00 | 1.00 | 1.00 | 0.90 | 1.00 |
| 3 | small-model | 0.06 | 0/1 | 0.00 | 0.00 | 0.00 | 0.46 | 0.00 |

Development

pip install -e ".[dev]"
pytest
ruff check src tests

MIT licensed.