gradetrail 0.1.2

A distributed LLM evaluation harness: declarative, cached, reproducible.

gradetrail

A distributed LLM evaluation harness that treats evals like tests: declarative, cached, reproducible.

Quickstart

pip install gradetrail
export ANTHROPIC_API_KEY=sk-ant-...

Define an eval:

# gsm8k_subset.yaml
name: gsm8k-subset
dataset:
  path: data/gsm8k_subset.jsonl
prompt: |
  Solve the following math problem. End your response with the line
  "Answer: <number>".

  {{ question }}
model:
  provider: anthropic
  name: claude-sonnet-4-6
scorer:
  type: regex
  pattern: 'Answer:\s*{{ answer }}\s*$'

data/gsm8k_subset.jsonl is three JSONL rows, one sample per line:

{"id": "1", "question": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", "answer": "72"}
{"id": "2", "question": "Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?", "answer": "10"}
{"id": "3", "question": "James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?", "answer": "624"}

Run it:

gradetrail run gsm8k_subset.yaml

Output from an actual cold run of this exact spec:

Samples: 3 (scored=3, provider_error=0, judge_error=0)
Mean score: 1.0000
Tokens: 189 in / 213 out
Cost: $0.0038
Cache hits: 0/3
Wall time: 2.50s

Run it again and it completes in about 40ms at $0.00: every response is cached, keyed on the request, not the scorer. The run writes results/<name>-<identity-hash>/results.jsonl (per-sample scores and responses) and manifest.json (spec hash, dataset hash, git SHA, timings, cost, so you can tell later exactly what produced a given number).

Why

• Cached: responses are keyed on (provider, model, base_url, resolved prompt, params); re-runs are free, and a prompt edit invalidates only the affected samples.
• Reproducible: every run writes a manifest (spec identity hash, dataset hash, judge file hash, requested vs served model, git SHA, gradetrail version).
• Multi-provider: Anthropic, OpenAI, and any OpenAI-compatible endpoint (vLLM, local inference servers), through one provider abstraction with a shared retry and backoff policy.
• Versioned judges: LLM-as-judge prompts are separate, hashed files, not strings inlined in the spec.
• Distributed: the same spec runs unchanged on a local asyncio backend or a single-machine Ray cluster; the backend is a CLI flag, not a spec field.

Prior art

Established tools cover much of this space: lm-evaluation-harness for academic benchmarks, Inspect for safety evaluations, promptfoo for application testing. gradetrail differs in three specific choices: the response cache is keyed independently of the scorer, so changing how you grade re-scores for free without re-calling the API; every run's identity is a hash over spec, dataset content, and judge file, so two result sets are comparable exactly when their hashes match; and the local and Ray backends share one spec format and one per-sample pipeline, so distribution is an execution detail rather than a rewrite.

Benchmark

500 samples, GSM8K test split, claude-sonnet-4-6, temperature 0, max_tokens: 512. Roughly 46.6k input / 56k output tokens per cold run.

Run	Backend	In-flight requests	Wall time	Cost	Accuracy
Cold	local, concurrency 8	8	176.6s	$0.98	97.2%*
Re-score after scorer fix	local, all cached	n/a	0.65s	$0.00	97.8%
Cold	ray, 8 workers	64 (8 workers x concurrency 8)	32.6s	$0.98	97.6%
Warm	ray, 8 workers	n/a	4.15s	$0.00	97.6%

* 97.2% was measured before a scorer bug fix (see the failure audit below); 97.8% and 97.6% are post-fix.

Three things worth being explicit about, since a benchmark table invites the wrong conclusions if read too quickly:

1. The ray-vs-local wall-time difference is a concurrency comparison, not framework magic. Ray ran 64 requests in flight (8 workers, each applying concurrency: 8 independently) while local ran 8, both against the same API rate limit. Ray's actual value here is not raw speed, it is that the identical spec ran on a different execution backend with a one-flag change; nothing in the spec or scoring logic had to know or care.
2. The $0.00 re-score row is the reason the cache exists. After fixing a scorer regex bug (below), re-measuring all 500 samples against the corrected pattern took 0.65s and zero API calls, because the cache is keyed on (provider, model, prompt, params), not on the scorer. Changing how you score never invalidates what you already paid to generate.
3. Temperature 0 does not guarantee identical outputs across runs. The local run measured 97.8% and the Ray run measured 97.6% on what should be the same 500 completions; the difference is one sample, consistent with ordinary API-level nondeterminism at temperature 0, not a bug in either backend.

Failure audit

97.2% understated the model's true accuracy by roughly 0.6 points due to measurement error, not model error. All 14 zero-score samples from the cold local run were inspected by hand:

• 3 were scoring artifacts: the model wrote a trailing period after the answer ("Answer: 25."), which the regex did not tolerate. Fixed.
• 1 was a dataset-convention mismatch: the model gave a more precise decimal answer than GSM8K's integer ground truth. Left as-is rather than loosened into fuzzy matching.
• 2 were truncations at the 512-token output ceiling, indistinguishable from a wrong answer without inspecting the raw response.
• 8 were genuine model reasoning errors.

Only the last 8 reflect the model actually being wrong.

Cross-provider check

The same 50 GSM8K samples, the same spec, one field changed (provider and name). This is an illustrative 50-sample check, not a full benchmark:

Model	Accuracy	Cost
claude-sonnet-4-6	96%	~$0.01
gpt-4o-mini	86%	~$0.006

Swapping providers is a two-line spec edit; the cache, scorer, and results format are identical across both runs.

Scorers

Three scorer types, one per eval, with worked examples in examples/:

• exact: string match against a sample field, with optional normalization (strip, lower, collapse whitespace).
• regex: a per-sample template-rendered pattern.
• judge: LLM-as-judge, with the judge prompt kept as a separate, versioned, hashed file.

Exit codes

• 0: at least one sample scored.
• 1: zero samples scored, or the run was aborted early (e.g. after repeated identical fatal errors such as a missing API key).

Roadmap

• Local async runner with caching and cost tracking
• Ray execution backend
• Surface stop_reason in per-sample results, so truncation at the token ceiling is distinguishable from a wrong answer (motivated by the failure audit above)
• Cache judge calls (keyed on response, judge file hash, and judge model) so judge-eval re-scores are also free
• Multi-turn and tool-use evals
• HuggingFace dataset loader (dataset-specific conversion scripts exist today; no first-class loader in the spec yet)

Design

See docs/design/eval-spec.md for the spec schema and semantics. A writeup of debugging an intermittent SQLite WAL race under Ray is in (https://chitvanpatel.com/blog/sqlite-wal-race-under-ray.html).

MIT license.