Skip to main content

trajtest — pytest for agent trajectories

Assert on what an agent did, not just what it said.

trajtest runs an agent against a mock world (in-memory FS, mock HTTP, KV store) and lets you assert on its trajectory — which tools it called, in what order, with what arguments, and what side effects it left behind. Deterministic. Local-first. No LLM-as-judge. CI-ready.

from trajtest import Expect, World, run

def my_agent(task, tools):
    rows = tools.http_get("https://api.example.com/rows").json()["rows"]
    tools.fs_write("/out/report.md", f"total: {sum(rows)}")
    return "done"

def test_agent_saves_before_claiming():
    world = World()
    world.http.mock("GET", "https://api.example.com/rows", json={"rows": [1, 2, 3]})

    traj = run(my_agent, "summarize the rows", world, max_steps=15)

    (Expect(traj)
        .called("http_get", url__contains="api.example.com")
        .then("fs_write", path="/out/report.md")
        .never_called("fs_delete")
        .completed_within(15)
        .finished()
        .world_file("/out/report.md", equals="total: 6")  # the world, not the narration
        .assert_all())

Why

Prompt-tests can't see the four ways agents actually fail:

  1. Confident false reports — the agent says "saved the file" but never called write. world_file() checks the mock world's final state, not the narration.
  2. Silent policy violations — narration says "I'll be careful", tool calls did a delete. never_called("fs_delete") catches it.
  3. Evidence-free passes — 94% pass rate, nobody knows what was verified. Every EvalFailure lists each assertion's outcome plus the full trajectory transcript. An eval with zero assertions fails loudly.
  4. Meaningless coverage — green suite, dangerous tools never exercised. traj.unused_tools(TOOLS) flags what the agent never touched.

Install

Not on PyPI yet; install from a clone of this repo:

pip install .

Core concepts

World — the mock environment. Strict by design: missing files raise FileNotFoundError, unmocked HTTP raises UnmockedRequestError. A world that forgives everything produces agents that pass evals and fail in prod.

  • world.fs — in-memory filesystem (read/write/append/delete/ exists/listdir/glob; str or bytes; ** glob support)
  • world.http — mock HTTP (mock(method, url, status=..., json=..., text=..., error="timeout"|"connection", times=n); exact then * wildcard matching; repeated mocks form a consumable queue — mock a 500 then a 200 to test retry logic)
  • world.kv — key-value scratch space; world.clock — controllable clock
  • world.reset() — wipe everything between scenarios

run(agent, task, world, max_steps=25, seed=None) — the agent is a plain function (task, tools) -> str. Every tool call is recorded (name, args, result, duration, errors). Tool errors are recorded and re-raised, so the agent can react to them. tools.note("...") captures narration separately — notes are never steps and never pollute sequences, so you can compare what the agent said against what it did. Crashes become traj.finished=False with traj.failure set; exceeding max_steps is a trajectory failure, not a hang.

sample(agent, task, world_factory, n=5) — for stochastic agents: n runs against fresh worlds. Seeds are passed when the agent accepts a seed keyword. Assert consistency with a loop:

for t in sample(agent, "task", make_world, n=5):
    Expect(t).never_called("fs_delete").completed_within(15).assert_all()

Expect(traj) — the assertion DSL. All methods chain; assert_all() reports every failure at once (fail-collecting, not fail-fast):

Assertion Meaning
called(name, *args, **kwargs) ≥1 call matching (positional prefix, kwarg subset, matchers ok)
called_times / at_least / at_most call counts
never_called(name, ...) forbids calls (optionally arg-scoped)
in_order(a, b, c) ordered subsequence, gaps allowed
then(name, ...) fluent chain: must occur after previous then
then_immediately(name, ...) strict adjacency required
first / last boundary calls
completed_within(n) / finished() step budget / clean completion
no_repeated_calls(max_repeats=2, name=None) loop detector: fails if an identical (tool, args) call happens more than max_repeats times; different args don't count
final_answer(matcher-or-value) the agent's return value
notes_contain / notes_match narration assertions
world_file(path, equals/contains/matches) final world state — the anti-lying assertion
world(desc, predicate) / custom(desc, fn) escape hatches

Kwarg matching uses parameter names, so called("http_get", url__contains="api.example.com") works whether the agent passed the URL positionally or by keyword. Suffix sugar: __contains __regex __startswith __endswith __gt __lt __ge __le. Matchers (ANY, Contains, Regex, StartsWith, EndsWith, Gt/Lt/Ge/Le, InstanceOf, Predicate, DictSubset) compose anywhere.

load_trace(path) — assert on a real recorded run. Reads a redbg trace (.jsonl or .jsonl.gz; stdlib only, redbg is not imported): each tool call/result pair becomes a call with its args as keyword args, the last response becomes the final answer, notes stay notes, and a session_end status of ok means finished. A call with no recorded result gets error="no result recorded". Call, order and answer assertions work; world_file / world raise EvalFailure because a recording has no mock World.

traj = load_trace("run.jsonl")
Expect(traj).in_order("read_file", "write_file").never_called("delete_file").assert_all()

traj.save_trace(path) — the reverse: write a trajectory as a redbg trace (.gz suffix gzips it), so a failing eval can be replayed with redbg replay or compared with redbg diff. Args are written by parameter name; outputs are the recorded result reprs (truncated at 500 chars), so the file is for debugging, not a lossless copy of the mock world.

pytest plugin — a world fixture (fresh World per test) is auto-registered. CLI — trajtest init scaffolds an example eval; trajtest run <path> runs evals through pytest.

Limitations (honest)

  • trajtest can't tell you your assertions are meaningful. Expect(t).finished().assert_all() is a weak eval and it will pass. Assert on trajectories and world state, not vibes.
  • The mock world is not the real world. No network latency, no real filesystem permissions, no file metadata (size/mtime), no concurrent access. Fidelity gaps are where false confidence breeds — keep mocks strict and review them.
  • Deterministic assertions vs stochastic agents is an inherent tension. Use sample(), seeds, and order-flexible assertions. If your agent is wildly non-deterministic, trajectory evals will be brittle — that's a signal about the agent, not just the eval.
  • Single-threaded. Agents that spawn threads or subprocesses calling tools concurrently are not supported.
  • No framework adapters in v1. The agent is (task, tools) -> str; wrapping a LangChain/CrewAI loop takes ~5 lines on your side.
  • No LLM-as-judge, by design. Semantic judgments belong in a custom predicate, not in the core.

License

MIT

Metadata

Release files for trajtest 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trajtest 0.1.1
File Size Uploaded Provenance Actions
trajtest-0.1.1.tar.gz 35.3 kB Without provenance Details for trajtest-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trajtest 0.1.1
File Interpreter ABI Platform Actions
trajtest-0.1.1-py3-none-any.whl Python 3 none any Details for trajtest-0.1.1-py3-none-any.whl

Total release size: 60.5 kB

Release files / trajtest-0.1.1.tar.gz

Download URL trajtest-0.1.1.tar.gz
Size 35.3 kB
Tags Source
SHA-256 checksum
How to use checksums
88013acee549d73d30ae5cf11e8665595e47c7b2c8051d383823054dc301fa17
BLAKE2b-256 checksum
How to use checksums
7644b72cf989edaf62e1a79d6cd47db655e50dba2df91f1aff01039444e84fa1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / trajtest-0.1.1-py3-none-any.whl

Download URL trajtest-0.1.1-py3-none-any.whl
Size 25.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f565856ec37b07c8817481d6223ef68b5a7d52c5987e80def20a5f88653ba161
BLAKE2b-256 checksum
How to use checksums
1b40a842fda0355486ab795564ec0a1106009b467c5749ae12919c50b8c8ee7a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for trajtest 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page