llm-code-benchmark

AI Cost Tracking

• 🤖 LLM usage: $2.7790 (45 commits)
• 👤 Human dev: ~$1030 (10.3h @ $100/h, 30min dedup)

Generated on 2026-07-19 using openrouter/qwen/qwen3-coder-next

Private benchmark repository for selecting OpenRouter models for repair-agent and validator-agent.

The benchmark never stores API keys. Paid runs require OPENROUTER_API_KEY from the environment or GitHub Actions Secrets. Standard CI and catalog-smoke do not make paid model requests.

python -m pip install -e ".[test]"
python -m pytest tests -q
llm-code-benchmark catalog-smoke --max-models 2

Paid smoke benchmark, after explicit approval and budget setup:

llm-code-benchmark live-smoke --max-models 2 --max-tasks 2 --repetitions 1 --budget-usd 0.25

Recommended next smoke uses explicit models instead of router auto-selection:

llm-code-benchmark live-smoke --model meta-llama/llama-4-scout,deepseek/deepseek-v4-flash --max-models 2 --max-tasks 2 --repetitions 1 --budget-usd 0.25 --dry-run

Remove --dry-run only after explicit paid-run approval.

Reports are written to reports/latest/ and immutable snapshots under reports/history/. Repair schema validation is intentionally separate from semantic patch validation, so path mismatches and invalid diffs are reported as repair task statuses rather than JSON schema failures.

Repair Output Modes

Repair benchmark requests now prefer file_edits: the model returns repository-relative paths and complete final file contents, and the benchmark generates a deterministic unified diff locally before validation and git apply --check. Native patch output remains supported and is scored separately through native_patch_success_rate; structured edits are tracked through structured_edit_success_rate. search_replace, create, and delete are not enabled in this version.

License

Licensed under Apache-2.0.