benchrail-ai 0.1.0

Context-derived, calibrated, evolving evals for LLM/agent systems

Benchrail

Know whether your AI agent actually got better.

Benchrail is a framework for building evals for LLM and agent systems. You change a prompt, a skill, a tool, a model — and today you have no trustworthy signal for whether the system improved against intended output. Eyeballing a few runs is anecdote, and non-determinism means "it looked better" might be luck. Benchrail gives you a domain-true standard to measure against.

Benchrail gives AI teams two things they need during change: a benchmark that defines quality and a release rail that prevents regressions from slipping through.

Research grounding

Benchrail is opinionated, but not "trust me bro" opinionated. The design follows several findings from AI evaluation research:

• Use multiple, context-specific metrics. Stanford CRFM's HELM argues for evaluating language models across scenarios and metrics beyond accuracy, including calibration, robustness, fairness, bias, toxicity, and efficiency. Benchrail applies that idea at the product level: each AI system gets its own benchmark and quality dimensions.
• Do not rely on public leaderboard scores as product truth. The Leaderboard Illusion documents how public model rankings can be distorted by selective disclosure, asymmetric data access, and overfitting to leaderboard dynamics. Benchrail is built around private, customer-owned evals tied to the actual system under test.
• LLM judges are useful, but only with calibration. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena shows strong LLM judges can approximate human preferences, while also noting biases such as position, verbosity, and self-enhancement. Benchrail therefore treats judges as candidates until they clear human-label calibration.
• Evaluation belongs in the lifecycle, not just the launch checklist. The NIST AI Risk Management Framework frames trustworthy AI work through governance, mapping, measurement, and management. Benchrail mirrors that lifecycle with benchmark history, calibration freshness, regression checks, and failure-driven evolution.
• Nondeterministic systems need repeated measurement. OpenAI's HumanEval paper helped popularize sampling-aware evaluation for code generation. Benchrail carries that lesson into product evals with multi-sample change evaluation and explicit inconclusive states.

The idea

Most eval tools make you hand-author every metric, then leave the eval set to rot as your product and business evolve. Benchrail instead treats the eval suite as a calibrated, evolving grading system derived from your business context:

declare context → derive the right rubric → compile + calibrate judges → run → evolve from discovered failures

• Domain-derived rubrics. Declare your business context (what the agent does, what "good" means, what failure modes matter) and Benchrail reasons out the correct set of grading criteria for it. The rubric a D2C company needs is not the rubric a logistics company needs — and a generic "accuracy" metric misses both.
• Calibrated judges. Each rubric compiles into LLM judges that are validated against a human gold set (agreement metrics, not vibes). A judge that can't beat the agreement bar doesn't get to gate anything.
• Evolves from real failures. The derived rubric is a prior — a hypothesis about what matters. Discovered production failures are the posterior that corrects it. Write evaluators for failures you discover, not ones you imagine.
• Judge-drift aware. Calibration isn't one-and-done; Benchrail watches for judges drifting out of agreement over time.

Status

Early, but the wedge is now runnable. The first slice proves domain derivation end to end: a declared Context Spec → an LLM-reasoned Rubric (with a per-criterion why traced to the spec) → a measured check that two domains genuinely diverge.

The product spine is now a custom benchmark lifecycle: define intended output, create cases, run a baseline, compare changes, and evolve the benchmark from real failures.

Use uv run benchrail help for the command list, or uv run benchrail help evaluate-change for command-specific options.

Install

Benchrail is in public-alpha shape. The PyPI distribution name is benchrail-ai; the CLI command and Python import package are benchrail.

From a clone:

git clone https://github.com/JeffBrines/benchrail.git
cd benchrail
uv sync
uv run benchrail help

Once published to PyPI:

uvx --from benchrail-ai benchrail help

Public Alpha Expectations

• APIs and artifact schemas may change before a stable release.
• Benchmark artifacts can contain sensitive prompts, policies, customer inputs, retrieved context, and model outputs; do not commit secrets or unredacted PII.
• LLM judges should not gate releases until they pass human-label calibration.
• Public leaderboard scores are not a substitute for product-specific evals.

Quick Example

Run the checked-in support-agent example:

uv run benchrail run examples/support-agent/benchmark.yaml \
  --outputs examples/support-agent/baseline.outputs.yaml \
  --system-version baseline \
  --out examples/support-agent/baseline.run.yaml

uv run benchrail run examples/support-agent/benchmark.yaml \
  --outputs examples/support-agent/candidate.outputs.yaml \
  --system-version candidate \
  --out examples/support-agent/candidate.run.yaml

uv run benchrail compare \
  examples/support-agent/baseline.run.yaml \
  examples/support-agent/candidate.run.yaml

Expected result: improved.

Create a benchmark

uv sync
uv run benchrail init "Support Agent" \
  --intended-output "Resolve customer issues accurately using company policy." \
  --good "Uses the current policy source" \
  --bad "Invents refund promises" \
  --user "Customers" \
  --risk "Wrong billing guidance" \
  --calibration-max-age-days 30 \
  --calibration-min-labels 4

This creates benchmarks/support-agent/benchmark.yaml, including the benchmark's release-gate calibration policy. Commands such as inspect, next, and evaluate-change use that policy by default; CLI policy flags override it for a single run.

Add a real case:

uv run benchrail add-case benchmarks/support-agent/benchmark.yaml \
  --name "Refund outside window" \
  --input "Can I get a refund after 60 days?" \
  --expected-behavior "Use the refund policy and do not invent exceptions." \
  --tag policy \
  --source production-failure

This benchmark artifact is the durable optimization target. Scorers, baseline runs, comparisons, and failure-driven evolution build on it.

Derive candidate scorers from the benchmark:

export OPENAI_API_KEY=sk-...
uv run benchrail derive-scorers benchmarks/support-agent/benchmark.yaml \
  --provider openai

Generated scorers are saved into the benchmark file as human-reviewable proposals. They are not release gates until they have been reviewed and, for LLM judges, calibrated against human labels.

Create a run from saved system outputs:

# outputs.prompt-v1.yaml
outputs:
  - case_id: refund-outside-window
    output: "Refunds are only available within 30 days."

Or collect outputs by running your system once per case:

uv run benchrail collect benchmarks/support-agent/benchmark.yaml \
  --command "python agent.py --input {input}" \
  --out outputs.prompt-v1.yaml

Command placeholders include {case_id}, {input}, {context}, and {expected_behavior}. Benchrail shell-quotes placeholder values, so do not wrap placeholders in additional quotes.

Or import outputs from an AI product/integration log:

{"case_id":"refund-outside-window","output":"Refunds are only available within 30 days.","trace_id":"trace-123","provider":"openai","model":"gpt-small","latency_ms":321.5,"total_tokens":162,"cost":0.0042,"feedback":"thumbs_down","outcome":"ticket_reopened"}

uv run benchrail import-outputs benchmarks/support-agent/benchmark.yaml \
  app-outputs.jsonl \
  --out outputs.prompt-v1.yaml

import-outputs accepts JSONL, CSV, YAML, or JSON. By default it expects case_id and output fields, validates case IDs against the benchmark, and writes the same outputs.yaml format used by benchrail run. It also preserves optional production metadata such as source_id, trace_id, timestamp, environment, provider, model, latency_ms, input_tokens, output_tokens, total_tokens, cost, feedback, feedback_score, outcome, and a scalar metadata mapping. Use --case-id-field and --output-field when your integration log uses different field names.

Run LLM judge scorers automatically:

export OPENAI_API_KEY=sk-...
uv run benchrail run benchmarks/support-agent/benchmark.yaml \
  --outputs outputs.prompt-v1.yaml \
  --system-version prompt-v1 \
  --judge-provider openai \
  --out runs/support-agent/prompt-v1.yaml

You can also provide manual scorer results in the outputs file. Manual results override judge execution for that scorer:

outputs:
  - case_id: refund-outside-window
    output: "Refunds are only available within 30 days."
    scorer_results:
      - scorer_id: policy-correctness
        status: pass
        rationale: "Matches the refund policy."

Unreviewed LLM judges are advisory by default. Their results are stored on run artifacts, but hard-gate summaries only count LLM judges whose scorer trust_status has been promoted to trusted after human review/calibration.

Calibrate judge scorers against human gold labels:

# labels.support-agent.yaml
labels:
  - case_id: refund-outside-window
    scorer_id: policy-correctness
    expected_status: pass
    notes: "Human reviewer confirmed the answer follows the refund policy."

uv run benchrail calibrate benchmarks/support-agent/benchmark.yaml \
  --run runs/support-agent/gold-run.yaml \
  --labels labels.support-agent.yaml \
  --out calibrations/support-agent/gold-run.yaml \
  --dry-run

Remove --dry-run to update each LLM judge scorer's trust_status to trusted, warning, or blocked based on sample size, TPR, TNR, and Cohen's kappa thresholds. Calibration reports include case-level failures showing false positives, false negatives, and missing judge predictions for human review. Saved reports are durable artifacts under calibrations/<benchmark-id>/.

Inspect calibration history and latest drift:

uv run benchrail calibration-history benchmarks/support-agent/benchmark.yaml

Inspect release-gate readiness with a freshness policy:

uv run benchrail inspect benchmarks/support-agent/benchmark.yaml \
  --calibration-max-age-days 30 \
  --calibration-min-labels 4

An LLM hard gate is only "ready" when it is trusted, has calibration provenance, has enough human labels, and its calibration is fresh under the policy.

Compare a candidate run against a baseline:

uv run benchrail compare \
  runs/support-agent/prompt-v1.yaml \
  runs/support-agent/prompt-v2.yaml

Or let Benchrail run the full agent-native change loop:

uv run benchrail evaluate-change benchmarks/support-agent/benchmark.yaml \
  --baseline-command "python agent.py --input {input}" \
  --candidate-command "python agent.py --input {input}" \
  --judge-provider openai \
  --artifact-dir evaluations/support-agent/current-change \
  --baseline-model gpt-premium \
  --candidate-model gpt-small \
  --baseline-cost 1.00 \
  --candidate-cost 0.25 \
  --samples 3

This writes baseline/candidate outputs, run artifacts, and an evaluation.yaml decision artifact with pass, fail, mixed, or inconclusive. By default, evaluate-change requires LLM hard gates to be release-ready under the benchmark's calibration policy. Use --allow-unready-gates only for exploratory runs. For nondeterministic agents, --samples repeats the baseline/candidate comparison and returns inconclusive when sample decisions disagree.

If you provide model and cost metadata, the evaluation artifact also records cost movement and a recommendation such as safe_to_downgrade, do_not_downgrade, needs_more_samples, or quality_improved_but_cost_increased. Costs are user-supplied estimates for now; use any consistent unit across baseline and candidate.

Evolve the benchmark from a real failure:

# failures/refund-60-days.yaml
input: "Can I get a refund after 60 days?"
bad_output: "Sure, I can refund that."
expected_behavior: "Use the 30-day refund policy and do not invent exceptions."
source: production-failure
tags: [policy, regression]
rationale: "The agent promised a refund outside policy."
name: "Refund outside window"

uv run benchrail --json evolve benchmarks/support-agent/benchmark.yaml \
  --from-failure failures/refund-60-days.yaml \
  --dry-run

Remove --dry-run to add the case and append an evolution history event.

Import an existing eval definition:

# external-eval.yaml
name: Support Refund Eval
intended_output: Answer refund questions using policy.
provider: json
external_id: support/refund-policy
cases:
  - id: refund-60-days
    name: Refund after 60 days
    input: Can I get a refund after 60 days?
    expected_behavior: Say refunds are only available within 30 days.
    source: imported

uv run benchrail import-json external-eval.yaml \
  --out benchmarks/support-refund/benchmark.yaml

Discover available and planned import/attach routes:

uv run benchrail --json discover "refund policy support"

Attach a run executed outside Benchrail:

uv run benchrail attach-run benchmarks/support-refund/benchmark.yaml \
  --provider inspect \
  --external-id run-123 \
  --system-version prompt-v1 \
  --summary external-run-summary.yaml \
  --out runs/support-refund/inspect-run-123.yaml

Run the milestone demo

The demo derives a rubric for the same agent task ("monthly revenue forecast") from two different Context Specs — D2C ecommerce vs freight logistics — and asserts the rubrics diverge.

uv sync
export ANTHROPIC_API_KEY=sk-ant-...
uv run python scripts/derive_demo.py --provider anthropic

Or run the same derivation through OpenAI/GPT:

export OPENAI_API_KEY=sk-...
uv run python scripts/derive_demo.py --provider openai

It writes out/rubric.d2c.yaml and out/rubric.logistics.yaml, prints a divergence report, and exits non-zero if the rubrics don't diverge (which would mean the derivation isn't domain-sensitive). Offline tests: uv run pytest.

Provider defaults can also be set with environment variables:

export BENCHRAIL_LLM_PROVIDER=openai
export BENCHRAIL_LLM_MODEL=gpt-5.5
uv run python scripts/derive_demo.py

What's wired

Piece	Module
Context Spec schema + loader	src/benchrail/context_spec.py
Benchmark schema + loader	src/benchrail/benchmark.py
Benchmark CLI	src/benchrail/cli.py
Agent inspection + next action	src/benchrail/agent.py
Failure-driven evolution	src/benchrail/evolve.py
Agent-native evaluation loop	src/benchrail/evaluation.py
External benchmark import	src/benchrail/importer.py
Source discovery	src/benchrail/discovery.py
Scorer derivation	src/benchrail/scorer.py
LLM judge execution	src/benchrail/judge.py
Judge calibration	src/benchrail/calibration.py
Run + comparison artifacts	src/benchrail/run.py
Rubric / Criterion schema	src/benchrail/rubric.py
Failure-mode-enumeration prompt	src/benchrail/prompt.py
Provider-neutral LLM derivation	src/benchrail/derive.py, src/benchrail/llm.py
Divergence measurement	src/benchrail/divergence.py

Inspect runner integration is the next major slice — see below.

Design docs

• DESIGN.md — thesis, positioning, build-on decision, trust model.
• BRANDING.md — Benchrail naming decision and rename migration plan.
• CUTOVER.md — external repository, package, domain, and compatibility checklist.
• CHANGELOG.md — release history and public-alpha contents.
• RELEASE.md — release validation, publishing, and announcement checklist.
• ROADMAP.md — execution plan from provider-neutral derivation to calibrated runs.
• docs/agent-harness.md — how build agents should operate Benchrail.
• docs/derivation-v0.md — the Context Spec schema, the derivation method, and a worked D2C-vs-logistics example showing how the rubrics diverge.

Benchrail is intended to run on top of Inspect AI rather than reimplementing an eval runner.

Security note: benchmark artifacts can contain production inputs, policies, customer data, and model outputs. Treat them as sensitive repo artifacts, review imported eval definitions before running judges over them, and avoid committing PII or secrets.

License

MIT.