tourney

Local-first benchmarking runner for AI models. Run your prompts against the field.

tourney runs your own prompts and datasets against multiple model APIs (your keys, your machine) and measures quality, latency, cost, and token usage — with the statistical rigor researchers expect: bootstrap confidence intervals, pinned model IDs, seeds, config hashes, and full run metadata in a local SQLite database. No hosted service, no telemetry.

Install

pip install "tourney[dashboard]"      # or plain `pip install tourney` for CLI/library only

Five-minute quickstart

tourney init            # scaffolds benchmark.yaml + cases.jsonl
tourney run benchmark.yaml
tourney show latest --failures
tourney serve           # local dashboard at http://127.0.0.1:8355

benchmark.yaml:

name: math-word-problems
models:
  - provider: openai
    model: gpt-4o-2024-08-06          # pin dated IDs for reproducibility
  - provider: anthropic
    model: claude-sonnet-5
  - provider: openai_compat           # anything OpenAI-compatible: Ollama, vLLM, OpenRouter...
    model: llama3.1:8b
    base_url: http://localhost:11434/v1
defaults: {temperature: 0, max_tokens: 256, seed: 42}
prompt:
  system: "Answer with only the final number."
  user: "{{ question }}"              # Jinja2 over each case's input
dataset: {path: cases.jsonl}
graders:
  - {type: numeric, tolerance: 0.001}
run: {concurrency: 8, retries: 3, timeout_s: 60}

API keys come from environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, or a custom api_key_env) — never from config files.

math-word-problems · run 01J9X4… · $0.213
┌──────────────────────┬───────┬──────────────┬─────────┬─────────┬─────────┬────────┐
│ model                │ score │ 95% CI       │ p50 lat │ p99 lat │ cost    │ errors │
├──────────────────────┼───────┼──────────────┼─────────┼─────────┼─────────┼────────┤
│ claude-sonnet-5      │ 0.960 │ [0.89, 1.00] │   944ms │  2107ms │ $0.0920 │      0 │
│ gpt-4o-2024-08-06    │ 0.940 │ [0.86, 0.98] │   812ms │  2431ms │ $0.1210 │      0 │
└──────────────────────┴───────┴──────────────┴─────────┴─────────┴─────────┴────────┘

Why another eval tool?

• Local-first. Results live in ./.tourney/results.db (SQLite). Copy it, query it, commit exports.
• Reproducible. Config hash, tourney/Python versions, and git commit stored per run. Deterministic graders by default; stats are seeded.
• Cheap to iterate. Responses are content-address cached — rerunning an unchanged benchmark costs $0.
• CI-native. tourney run --json --fail-under 0.85 exits non-zero when a model regresses.
• Honest statistics. Bootstrap CIs over cases; errors score 0 instead of being dropped.

Graders

Multiple graders per benchmark average into the case score. LLM-as-judge is planned as strictly opt-in.

Library

import tourney

result = tourney.run("benchmark.yaml")
result.summary()      # per-model dicts: score, CI, latency percentiles, cost
result.to_records()   # per-completion rows -> pd.DataFrame(result.to_records())

CLI reference

tourney init                     scaffold a new benchmark
tourney run <config>             run it (--model, --limit, --no-cache, --json, --fail-under)
tourney list                     recent runs
tourney show <run|latest>        summary table (--failures for the failing cases)
tourney export <run> --format    jsonl | csv | md
tourney serve                    local dashboard (requires tourney[dashboard])

Extending

Third-party providers and graders register via entry points (tourney.providers, tourney.graders) — publish a package with a Provider or Grader subclass and it becomes available by name in configs. A mock provider ships in the box for dry runs and tests.

Development

pip install -e ".[dev]"
pytest        # fully offline — network is disabled in the test suite

Apache-2.0.