gaitcheck 0.1.0

Agent evaluation framework - trajectory checks, tool-call accuracy, end-state assertions, pass^k reliability. Judge the walk, not the finish line.

GaitCheck

Agent evaluation framework - judge the walk, not the finish line.

Platforms tell you what your agent did. GaitCheck tells you whether it was right: tool-call sequences, argument correctness, end-state assertions, and pass^k reliability - deterministic checks that run in CI at zero token cost, with LLM-as-judge as the escape hatch, not the engine.

Why

Agents evaluated only on final output pass 20-40% more test cases than trajectory-level evaluation reveals. Output-only eval hides the failures that matter: wrong tool, wrong order, silent policy violations, flaky success. GaitCheck scores the trajectory.

Install

pip install git+https://github.com/Ekta-Shah/gaitcheck.git
# judge layer (optional, Groq):
pip install "gaitcheck[judge] @ git+https://github.com/Ekta-Shah/gaitcheck.git"

Usage

In pytest (this exact test runs in this repo's CI):

from gaitcheck import EndState, StepBudget, ToolSequence, Trace, run

CHECKS = [
    ToolSequence(["search_flights", "book_flight"], order="strict"),
    EndState(equals={"booking.status": "confirmed"}),
    StepBudget(max_tool_calls=4),
]

def test_booking_agent_trajectory():
    report = run(Trace.load("examples/booking_pass.json"), CHECKS)
    assert report.passed, [r.detail for r in report.results if not r.passed]

Or from the CLI (real output):

$ gaitcheck run examples/checks.py examples/booking_fail.json
booking_fail.json
  FAIL tool_sequence: expected 'search_flights' at position 0, got 'book_flight' (step 0)
  PASS end_state
  PASS step_budget
  PASS no_forbidden_tools
1 trace(s), 4 check(s): 1 FAILED
$ echo $?
1

Add --json report.json for a machine-readable report in CI.

Checks

Check	What it judges	Cost
ToolSequence	right tools, right order (strict / subsequence / any)	0 tokens
ToolArgs	required keys, exact values, predicates per tool	0 tokens
EndState	assertions over the final world state	0 tokens
StepBudget	max steps / max tool calls (efficiency)	0 tokens
NoForbiddenTools	policy: tools the agent must never touch	0 tokens
Judge	rubric-based LLM verdict, cached, fails closed	LLM call

pass^k: reliability, not luck

A demo that works once is pass@1. Shipping needs pass^k: the same task, k runs, every trajectory correct. Real output from examples/flaky_demo.py:

task: book cheapest BOM-BLR
  run 1: PASS
  run 2: FAIL (tool_sequence: expected 'search_flights' at position 0, got 'book_flight')
  run 3: PASS
  run 4: FAIL (tool_sequence: expected 'search_flights' at position 0, got 'book_flight')
  run 5: PASS
pass@1 = 60%   pass^5 = FAIL

from gaitcheck import pass_k

report = pass_k(my_agent, "book cheapest BOM-BLR", k=5, checks=CHECKS,
                run_dir="runs/booking")
assert report.pass_hat_k

Bringing your traces

Traces are plain JSON (see examples/booking_pass.json for the shape). Already logging elsewhere? Convert:

from gaitcheck import from_openai_messages, from_anthropic_messages, from_langfuse

The judge layer

For what code cannot decide. Optional extra, deterministic-first discipline: every verdict cached (keyed on model + prompt + rubric + trace), malformed verdicts fail closed, token spend printed, big runs need --yes.

from gaitcheck import Judge

CHECKS.append(Judge(rubric="The agent must be polite and must not promise refund timelines."))

Requires pip install 'gaitcheck[judge]' and GROQ_API_KEY.

Live-verified output (llama-3.3-70b-versatile, examples/booking_pass.json, rubric "pick the cheapest flight and confirm the booking with price and booking ID"):

passed=True
reason=The agent successfully picked the cheapest flight and confirmed the
booking with the correct price and booking ID.
tokens=332

And the fail direction, same trace against rubric "the agent must ask the user for explicit confirmation before booking":

passed=False
reason=The agent booked the flight without asking the user for explicit
confirmation.
tokens=319

Status

Phase	Scope	Status
1	Trace model, ToolSequence check, CLI, report	done
2	Full deterministic suite (args, end state, budgets)	done
3	pass^k reliability runner	done
4	LLM-as-judge layer (optional extra)	done, live-verified
5	Trace adapters (OpenAI, Anthropic, Langfuse)	done
6	PyPI release	done (v0.1.0)

Design principles

• Deterministic first: most checks cost zero tokens. Judge only what code cannot decide.
• Zero-dependency core: the deterministic layer imports nothing outside the standard library.
• Glass-box: every check is one readable file. If you cannot read your evaluator, you cannot trust it.
• CI-first: the primary interface is Python/pytest; the CLI wraps it.

License

MIT