tourney

Local-first benchmarking runner for AI models. Run your prompts against the field.

tourney runs your own prompts and datasets against multiple model APIs (your keys, your machine) and measures quality, latency, cost, and token usage — with the statistical rigor researchers expect: bootstrap confidence intervals, pinned model IDs, seeds, config hashes, and full run metadata in a local SQLite database. No hosted service, no telemetry.

Install

Requires Python 3.11+.

pip install tourney                 # CLI + library
pip install "tourney[dashboard]"    # adds the local results dashboard (tourney serve)

Or with uv:

uv tool install tourney             # install the CLI globally
uv run --with tourney tourney --help   # or run it without installing

Five-minute quickstart

tourney init            # scaffolds benchmark.yaml + cases.jsonl
tourney run benchmark.yaml
tourney show latest --failures
tourney serve           # local dashboard at http://127.0.0.1:8355

benchmark.yaml:

name: math-word-problems
models:
  - provider: openai
    model: gpt-4o-2024-08-06          # pin dated IDs for reproducibility
  - provider: anthropic
    model: claude-sonnet-5
  - provider: gemini
    model: gemini-2.5-flash
  - provider: openai_compat           # anything OpenAI-compatible: Ollama, vLLM, OpenRouter...
    model: llama3.1:8b
    base_url: http://localhost:11434/v1
defaults: {temperature: 0, max_tokens: 256, seed: 42}
prompt:
  system: "Answer with only the final number."
  user: "{{ question }}"              # Jinja2 over each case's input
dataset: {path: cases.jsonl}
graders:
  - {type: numeric, tolerance: 0.001}
run: {concurrency: 8, retries: 3, timeout_s: 60}

API keys come from environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, or a custom api_key_env) — never from config files.

math-word-problems · run 01J9X4… · $0.213
┌──────────────────────┬───────┬──────────────┬─────────┬─────────┬─────────┬────────┐
│ model                │ score │ 95% CI       │ p50 lat │ p99 lat │ cost    │ errors │
├──────────────────────┼───────┼──────────────┼─────────┼─────────┼─────────┼────────┤
│ claude-sonnet-5      │ 0.960 │ [0.89, 1.00] │   944ms │  2107ms │ $0.0920 │      0 │
│ gpt-4o-2024-08-06    │ 0.940 │ [0.86, 0.98] │   812ms │  2431ms │ $0.1210 │      0 │
└──────────────────────┴───────┴──────────────┴─────────┴─────────┴─────────┴────────┘

Why another eval tool?

• Local-first. Results live in ./.tourney/results.db (SQLite). Copy it, query it, commit exports.
• Reproducible. Config hash, tourney/Python versions, and git commit stored per run. Deterministic graders by default; stats are seeded.
• Cheap to iterate. Responses are content-address cached — rerunning an unchanged benchmark costs $0.
• CI-native. tourney run --json --fail-under 0.85 exits non-zero when a model regresses.
• Honest statistics. Bootstrap CIs over cases; errors score 0 instead of being dropped. pass@k and pass^k when you sample more than once.

pass@k and pass^k

Sample each case more than once and report how often a model gets there at all versus how often it gets there reliably:

run: {repeats: 10}
report:
  pass_at_k: [1, 5]     # P(at least one of k attempts passes)
  pass_hat_k: [5]       # P(all k attempts pass)

pass@k uses the unbiased estimator from the Codex paper; pass^k is its hypergeometric counterpart. Both are computed per case and averaged over cases, and both need repeats >= k — configs that ask for more are rejected at load time rather than after a paid run.

A sample counts as passing only when every grader on it passed, so partial credit contributes to score but not to these columns. Repeats are cached independently, so distinct samples stay distinct; set temperature above 0 or every repeat will be the same answer and pass@k will equal pass@1.

Graders

Multiple graders per benchmark average into the case score. LLM-as-judge is planned as strictly opt-in.

Library

import tourney

result = tourney.run("benchmark.yaml")
result.summary()      # per-model dicts: score, CI, latency percentiles, cost
result.to_records()   # per-completion rows -> pd.DataFrame(result.to_records())

CLI reference

tourney init                     scaffold a new benchmark
tourney run <config>             run it (--model, --limit, --no-cache, --json, --fail-under)
tourney list                     recent runs
tourney show <run|latest>        summary table (--failures for the failing cases)
tourney export <run> --format    jsonl | csv | md
tourney serve                    local dashboard (requires tourney[dashboard])

Extending

Third-party providers and graders register via entry points (tourney.providers, tourney.graders) — publish a package with a Provider or Grader subclass and it becomes available by name in configs. A mock provider ships in the box for dry runs and tests.

Development

pip install -e ".[dev]"
pytest        # fully offline — network is disabled in the test suite

Apache-2.0.