crowdbench-run 0.1.3

Ephemeral, hash-pinned intelligence-benchmark runner for CrowdBench.

crowdbench-run

The ephemeral, hash-pinned, open-source intelligence-benchmark runner. Pure Python (httpx + psutil + platformdirs + stdlib; never torch/transformers — HTTP only), uv-managed, invoked via uvx identically on macOS/Linux/Windows.

It probes hardware/engine, normalizes config against the vendored packages/shared JSON artifacts, administers the benchmark item set against a locally-served OpenAI-compatible endpoint, captures per-item timing forensics, and uploads model outputs — scoring is entirely server-side and answer keys never touch this package. Resumable state lives in ~/.crowdbench/ (via platformdirs).

Dark-period invocation

No real version is published to PyPI until the repo flip; run from source.

Bare crowdbench-run on a terminal launches the guided wizard — a zero-file, zero-flag contribution flow (splash → one-time consent → detect engines → pick a model → pick suites by category → time budget + a live throughput estimate → reasoning → confirm → run + poll to a final state). It runs only what this build can administer (core-v1 and code-v1 directly); industry lm-eval suites are shown with their cost and the exact crowdbench-run bridge … command, never run under the wizard's pretense. On a non-TTY (piped/CI) the bare command prints usage and exits — it never hangs for input.

uvx --from ./packages/runner crowdbench-run            # the guided wizard (TTY)
uvx --from ./packages/runner crowdbench-run detect
uvx --from ./packages/runner crowdbench-run inventory
# The flag path (what the wizard drives; agents use it directly). --dataset is now OPTIONAL —
# omit it and the public prompts dataset is auto-fetched from the API into the content-addressed cache.
uvx --from ./packages/runner crowdbench-run core-v1-rc \
    --endpoint http://localhost:8080/v1 --model <model> --agree-tos --yes

The RUN flow is probe → confirm → run → upload: probe hardware + engine, confirm the plan (interactive, or --yes/--dry-run for agents), administer every item capturing output + per-item timing (capture requirement 12), then POST to /v1/runs/outputs, which returns a pre-issued run URL (status pending-score until the server scores). An interactive run shows the same ASCII splash as the wizard (TTY only — never under --json, --yes, or a pipe), and the benchmark id is validated before the consent screen, so an unsupported id bounces immediately rather than after you have agreed.

An interactive run adds three things (all TTY-only; --yes/--json surfaces are unchanged):

• Model picker. With several loaded models and no --model, the wizard's numbered picker runs — never a silent first-pick. Non-interactive stays documented behaviour: --model, else the endpoint's first loaded model.
• Pre-flight, before the proceed prompt. How many items against which model; an estimated duration on this machine from a short (<30 s) throughput probe — reasoning-aware: the estimate multiplies by the suite's reasoning multiplier when reasoning is configured on OR the probe observed native reasoning output (thinking models emit thousands of reasoning tokens per item); what uploads (outputs + timings), that you get a run URL, and that scoring is server-side (for code-v1: in a sandboxed executor, a few extra minutes after upload).
• Honest progress + safe abort. Progress lines read problem 12/100 · ~1h 40m left (rolling ETA from measured per-item times); raw item ids appear only with --verbose (they always ride the --json payload). Ctrl-C anytime — partial runs are never uploaded: SIGINT aborts cleanly (exit 130) and uploads nothing — a partial suite never scores, so it is never submitted, and a started suite is never time-truncated.

code-v1 runs the identical path: the runner administers the pooled code prompts and uploads OUTPUTS + timings, and the held-out tests never leave the server (they are the answer key — see executor/README.md). Because code-v1 is scored asynchronously (queue → ephemeral container → judge) rather than inline, the runner polls it to a final state on a longer per-benchmark window (15 min vs. core-v1's 5); if the window closes first it says so and tells you to report the run — it never assumes success. code-v1 uses the same sampler defaults as core-v1 (temperature 0, fixed seed): no in-repo code-v1 spec prescribes a different profile, and the runner does not invent one. Consent is shown once (state in ~/.crowdbench); --agree-tos attests a human has seen and agreed to the terms (what uploads; the public CC-BY-NC-SA 4.0 dataset; immutability) and is required to upload non-interactively.

Commands

command	what it does
(no arguments, on a TTY)	the guided wizard — zero-file, zero-flag contribution flow
<benchmark-id>	run one benchmark end-to-end (the flag path); core-v1 / core-v1-rc / code-v1 / throughput-std-v1 in this phase
bridge lmeval:<task>:<variant>	run/parse an allowlisted lm-eval suite → validate → upload
detect	scan common local serving ports; report apps + loaded models (by route signature)
inventory	list local models (Ollama / LM Studio / HF cache / user dirs) + benchmark tooling
doctor	probe + connectivity + state report for support
cache list|clear	manage the content-addressed dataset cache

Key flags (dual-surface parity — every wizard prompt has a flag): --endpoint, --api-base, --dataset (omit to auto-fetch), --model, --model-path, --reps N, --seed, --reasoning-effort, --reasoning-budget, --agree-tos, --minutes, --hf-repo/--hf-revision/--unattributed, --endpoint-api-key (env CROWDBENCH_ENDPOINT_API_KEY; never logged/uploaded/stored), --dry-run, --no-upload/--save/--upload-only, --json, --yes, --verbose (raw item ids on the progress lines), --sudo-probes, --ports. Auth uses CROWDBENCH_API_KEY (else an anonymous submitter is minted and cached).

Trust posture

• Outputs only. The runner uploads model outputs + telemetry; it never sees, computes, or transmits answer keys. Benchmark item content is inert data — never a tool, never agentic.
• Dataset integrity. A dataset's sha256 must match the benchmark's runner_spec pin before every run (capture requirement 11); a mismatch refuses to run (a tampered set never runs).
• Vendored artifacts, byte-matched. crowdbench_run/_artifacts/*.json are copied verbatim from packages/shared/artifacts so both languages read one source; CI fails on drift (scripts/check_artifact_drift.py — the third leg of the cross-language drift check).
• Endpoint key privacy. --endpoint-api-key is forwarded as the target endpoint's Authorization header only, never logged/uploaded/stored.

Supply-chain audit trail: packages/runner/BUILD-TIME-CHECKS.md.

Publishing (PyPI Trusted Publishing)

Releases go out through .github/workflows/publish.yml using PyPI Trusted Publishing (OIDC) — there is no long-lived PyPI API token in this repo, ever. GitHub mints a short-lived OIDC token per run and pypa/gh-action-pypi-publish (pinned by full commit SHA) exchanges it with PyPI. Three human gates stand between a dispatch and a live release: workflow_dispatch only (no push/tag/schedule), a typed confirm input that must equal publish-to-pypi, and environment: pypi (owner approval before any publish step runs). A preceding job builds the sdist+wheel, runs twine check --strict, install-smokes both artifacts, and refuses the placeholder 0.0.x version line.

Owner one-time setup — verify on pypi.org. Create the Trusted Publisher for this project under Manage project → Publishing (or Your projects → Publishing for a pending publisher). These three fields must match the workflow exactly, or PyPI rejects the OIDC exchange:

PyPI Trusted-Publisher field	Must be
PyPI project (package) name	crowdbench-run
GitHub repository + workflow filename	crowdbench-ai/crowdbench-dev, workflow publish.yml
Environment name	pypi

(The owner also configures the pypi GitHub Environment's protection rule — required reviewers — so gate #3 actually pauses for approval.) This PR only prepares the workflow and docs; wiring the Trusted Publisher and the environment protection rule is an owner action on pypi.org / GitHub, and nothing publishes during the dark period.

Development

cd packages/runner
python -m pip install -e ".[dev]"
python -m pytest -q

The test suite runs on ubuntu/macos/windows in CI (a release gate — cross-platform posture).

---

CrowdBench is a service of Hooch Labs LLC. © 2026 Hooch Labs LLC · Apache-2.0.