lucebench

Endpoint capability gate for OpenAI-compatible chat-completion servers — and Anthropic-messages tool scenarios. Ten evaluation areas, progressive snapshot levels, vendored fixtures, machine-readable reports.

Distribution, import, primary CLI, and GitHub repo are all lucebench (no hyphen). Hyphenated console scripts (luce-bench, …) remain as aliases.

Who this is for

Not this tool: research leaderboard campaigns (use lm-evaluation-harness / OpenCompass), app/prompt/RAG QA (Promptfoo, DeepEval), or full agent loops (fizeau, terminal-bench). lucebench scores the endpoint.

Areas: smoke, ds4-eval, gsm8k, truthfulqa-mc1, hellaswag, code, longctx, agent, agent_recorded, forge.

Install

PyPI status (2026-07-23): package name reserved in CI, but trusted publishing is not configured on pypi.org yet (invalid-publisher on tag runs). Use the git pin until docs/helix/05-deploy/runbook-pypi-trusted-publishing.md is completed.

# Current reliable install — pin a release tag
uvx --from "git+https://github.com/easel/lucebench@v0.3.2" \
  lucebench --base-url http://127.0.0.1:8000

# Unreleased main
uvx --from "git+https://github.com/easel/lucebench@main" \
  lucebench --base-url http://127.0.0.1:8000

# After PyPI is live (see runbook)
# uvx lucebench --base-url http://127.0.0.1:8000
# uv add lucebench
# pip install lucebench

anthropic is a hard runtime dependency (forge + default judge path). The legacy [forge] extra still resolves as an empty alias.

Every run prints a version banner first — [lucebench] vX.Y.Z — so stale caches are easy to spot.

Quick start

# Smoke (3 cases) — default when --areas is omitted
uvx --from "git+https://github.com/easel/lucebench@v0.3.2" \
  lucebench --base-url http://127.0.0.1:8000

# Progressive post-change gate (host + levels)
uvx --from "git+https://github.com/easel/lucebench@v0.3.2" \
  lucebench snapshot --level level1 \
  --base-url http://127.0.0.1:8000 --name my-machine

# OpenRouter
export OPENROUTER_API_KEY=sk-or-...
uvx --from "git+https://github.com/easel/lucebench@v0.3.2" \
  lucebench --base-url https://openrouter.ai/api \
  --model qwen/qwen3.6-27b --auth-env OPENROUTER_API_KEY

# Multi-area sweep (see "areas all" policy below)
uvx --from "git+https://github.com/easel/lucebench@v0.3.2" \
  lucebench --areas all --name my-machine \
  --base-url http://127.0.0.1:8000

--areas all policy

--areas all runs: smoke, ds4-eval, gsm8k, truthfulqa-mc1, hellaswag, code, longctx, agent, forge.

It does not include agent_recorded or agent_recorded_v1 (side effects: optional service restart, LLM judge cost). Run those explicitly:

lucebench --areas agent_recorded --base-url http://127.0.0.1:8000

snapshot --level level3 does include agent_recorded by design (baseline-grade full suite). See docs/helix/02-design/contracts/CONTRACT-001-cli.md.

More examples

# Single case, json-out for downstream analysis
lucebench --areas ds4-eval --case-id aime2025-02 \
  --base-url http://localhost:8080 --json-out /tmp/aime02.json

# Limit each area to N questions
lucebench --areas all --name quick --questions 2 \
  --base-url http://localhost:8080

# Parallel against a stateless gateway (skip on single-GPU local servers)
lucebench --areas ds4-eval --base-url https://openrouter.ai/api \
  --model openai/gpt-5.4 --auth-env OPENROUTER_API_KEY --parallel 8

# Single-case multi-mode reasoning probe
lucebench-probe --case-id aime2025-02 \
  --url http://localhost:8080 --out-dir ./probes/my-model

A sweep writes per-area JSON and a combined _summary.md under ./snapshots/<name>/. Envelope fields and headline metrics: docs/helix/02-design/contracts/CONTRACT-002-result-envelope.md and docs/metrics.md.

What's benchmarked

Each row in the result carries:

• pass (bool), graded (full grader output)
• wall_seconds, http_status, error
• prompt_tokens, completion_tokens, timings (when surfaced by the server)
• content, reasoning_content, finish_reason, finish_details

The default sampling shape is send-nothing-explicit — the server applies its own defaults. Sampling flags are omitted unless the user sets them.

Snapshot levels

lucebench snapshot --level level1 --base-url http://127.0.0.1:8000

Programmatic use

from lucebench.areas import ds4_eval
from lucebench.runner import run_case

cases = ds4_eval.load_ds4_eval_cases()
case = next(c for c in cases if c["id"] == "aime2025-02")

row = run_case(
    url="http://localhost:8080",
    case=case,
    model="my-model",
    think=True,
)
graded = ds4_eval.grade_case(case, row)
print(graded["pass"], graded["given"], "/", graded["correct"])

Library imports used by external sweeps are provisional until a library Contract lands; prefer the CLI for stable automation.

Attribution

This project redistributes evaluation fixtures from upstream MIT- licensed projects. See NOTICE for full attribution; in short:

• ds4-eval cases — antirez/ds4, MIT
• GSM8K cases — openai/gsm8k, MIT
• TruthfulQA MC1 cases — truthful_qa, Apache-2.0
• HellaSwag cases — Rowan/hellaswag, MIT
• HumanEval prompts — openai/human-eval, MIT
• forge eval scenarios — antoinezambelli/forge, MIT

The lucebench code itself is Apache-2.0.

Product docs

Governed product docs live under docs/helix/ (vision, competitive analysis, PRD, contracts). Alignment for external release: docs/helix/06-iterate/alignment-external-release.md.

Contributing

git clone https://github.com/easel/lucebench
cd lucebench
uv sync --extra dev
uv run pytest
uv run ruff check src tests scripts

CI runs the same matrix on Python 3.10–3.13 + a wheel-build check that verifies fixtures are bundled.