crowdbench-run 0.1.8

Ephemeral, hash-pinned intelligence-benchmark runner for CrowdBench.

crowdbench-run

The ephemeral, hash-pinned, open-source intelligence-benchmark runner. Pure Python (httpx + psutil + platformdirs + stdlib; never torch/transformers — HTTP only), uv-managed, invoked via uvx identically on macOS/Linux/Windows.

It probes hardware/engine, normalizes config against the vendored packages/shared JSON artifacts, administers the benchmark item set against a locally-served OpenAI-compatible endpoint, captures per-item timing forensics, and uploads model outputs — scoring is entirely server-side and answer keys never touch this package. Resumable state lives in ~/.crowdbench/ (via platformdirs).

Dark-period invocation

No real version is published to PyPI until the repo flip; run from source.

Bare crowdbench-run on a terminal launches the guided wizard — a zero-file, zero-flag contribution flow (splash → one-time consent → detect engines → pick a model → model identity → ONE numbered suite catalog grouped by category → reasoning → confirm → run + poll to a final state → wrap-up). Every available row in the catalog runs (0.1.8): the checks this build administers (core-v1, code-v1) through its own run path, and the industry lm-eval suites through its bridge — all means both, and each row's exact crowdbench-run bridge … command stays printed underneath for agents and scripts to copy. An industry suite is checked before it starts: lm-eval must be installed and on the accepted version list (said in plain English with the one command to run), and the benchmark must carry a pinned dataset revision — a suite that could never be accepted never begins. The coding category says out loud that it has no industry suite by design: every standard one scores by executing model-generated code on your machine, which CrowdBench never does. On a non-TTY (piped/CI) the bare command prints usage and exits — it never hangs for input.

Selection happens against the catalog's own numbering — there is no time-budget question and no second, renumbered "fits your budget" list (0.1.7, owner ruling). Greyed rows keep their numbers but are not selectable: picking one gets one explanatory line and a re-ask against the same numbers. Prompts that follow a long non-input phase (endpoint scan, speed measurement, fingerprinting) flush pending stdin first, so a key pressed while a spinner ran is never silently consumed as an answer; selections are echoed back in suite names.

Two things the wizard settles before it asks you anything else:

• Are the questions actually published? Each directly-runnable suite's item set is probed while the catalog is built (the same cached fetch the run itself uses, so the run pays nothing extra). A suite the server has not published is greyed with the reason and excluded — rather than failing after you have answered every prompt. A probe that merely fails (timeout, 5xx, offline) never hides a suite; only a definitive "not published" does.
• Which upload are these weights? One coherent identity step: a resolved repo is shown as org/name @ revision with its plain-English source and the local path; when only a local path is known, the wizard says exactly that — never a bare served name for yes/no "confirmation" — and looks the name up on Hugging Face straight away (0.1.8; --no-hf-search opts out). The lookup (the runner's one network target beyond crowdbench.ai and your endpoint, owner-approved) sends ONLY the served model name and fails soft in ~5 s. It offers exact name matches only: the hub searches by substring, so the raw results mix in different uploads wearing similar names (…-fp16-mtp, a distill). What survives differs only by org — which only you know — so nothing is ever auto-picked, and a picked result records identity_source: user_confirmed because you chose. No exact match is one calm line and an unattributed continue, not an interrogation. Non-interactive runs never search. The fingerprint then just runs, announced in one sentence (--no-fingerprint opts out): it is computed from your files locally, and only the fingerprint — never the files — is uploaded.

The times the wizard quotes follow the thinking level you pick. A suite's estimate is a non-thinking floor; choosing low/medium/high multiplies it by that suite's thinking cost, choosing off leaves it at the floor, and leaving it at model default shows a range — ~88s–24m, depends how much the model chooses to think — because a model that thinks by default lands near the top and one that doesn't lands at the floor, and nothing has decided which yet. When you pick off on an engine that itself reports thinking ON by default, the confirm screen warns that the request may be ignored and what the time can run toward — a warning only; the run still records what the model actually did.

uvx --from ./packages/runner crowdbench-run            # the guided wizard (TTY)
uvx --from ./packages/runner crowdbench-run detect
uvx --from ./packages/runner crowdbench-run inventory
# The flag path (what the wizard drives; agents use it directly). --dataset is now OPTIONAL —
# omit it and the public prompts dataset is auto-fetched from the API into the content-addressed cache.
uvx --from ./packages/runner crowdbench-run core-v1-rc \
    --endpoint http://localhost:8080/v1 --model <model> --agree-tos --yes

The RUN flow is probe → confirm → run → upload: probe hardware + engine, confirm the plan (interactive, or --yes/--dry-run for agents), administer every item capturing output + per-item timing (capture requirement 12), then POST to /v1/runs/outputs, which returns a pre-issued run URL (status pending-score until the server scores). An interactive run shows the same ASCII splash as the wizard (TTY only — never under --json, --yes, or a pipe), and the benchmark id is validated before the consent screen, so an unsupported id bounces immediately rather than after you have agreed.

The default held-out run is a short, server-issued set

A held-out check answers a server-issued subset (~20 questions), not the whole pool. The runner asks the API to deal it (POST /v1/benchmarks/{id}/challenge) and administers exactly the item ids it is dealt — it never samples a pool itself, because a client that picks its own "random" subset picks a favourable one. Scoring accepts only an answered set that matches an issued draw or the complete pool; anything else parks unscored.

• The challenge is requested as late as possible — after consent, after the confirm screen, immediately before the first question. A draw is consumed when its answers are scored, so a run you abort never burns one.
• --full answers the complete pool — the opt-in "thorough" run.
• Issuance failure is never silent (0.1.7). A transport failure or timeout retries (each retry deals a fresh draw; orphaned draws are harmless — scoring matches any issued challenge). Still failing, an interactive run stops and asks — try again (the default) / run the complete pool at its stated cost / abort — and a non-interactive (--yes) run refuses with a machine-readable error and non-zero exit: a timeout is not permission to multiply the workload. Only when the server says it deals no challenges (404/405 — an older API, or a benchmark that always runs complete) does the documented full-pool fallback apply, and interactively even that is an explicit choice. It never falls back to a locally-chosen subset.
• A pool no larger than one draw is run whole without asking (the server refuses a draw bigger than its pool rather than clamping it).

These suites are a validity check, not a public score. They exist because every industry-standard score is self-reported by the contributor's machine; the held-out questions are the one probe scored against answers nobody has seen. They place nothing on a leaderboard, and the runner's copy says so in plain words — user-facing text uses display names ("Reasoning & knowledge check", "Coding check") and never the internal ids, which stay unchanged in payloads, run URLs, and API paths.

An interactive run adds four things (all TTY-only; --yes/--json surfaces are unchanged):

• Model picker. With several loaded models and no --model, the wizard's numbered picker runs — never a silent first-pick. Non-interactive stays documented behaviour: --model, else the endpoint's first loaded model.
• Thinking level, before the confirm screen. A numbered choice — model default / off / low / medium / high — that feeds the same --reasoning-effort machinery the flag does and is recorded identically. off sends the upstream-documented reasoning_effort: none; support is model-dependent, so an engine may ignore or reject it and the run records what the model actually did (measured from its output), not what was asked. Non-interactive is unchanged: the flag, or the model's default.
• Pre-flight, before the proceed prompt. What the check is in plain words, how many questions against which model, and an estimated duration range measured on this machine. The estimate is a range, never a single confident number: its low end is the suite's token floor at the fastest rate measured, its high end that floor at the slowest rate, multiplied by the suite's thinking cost only when the probe actually observed reasoning (configuration alone is not evidence — an engine may ignore the control). The probe sends the selected model's id and discards its first generation, so a cold weight load never lands in the measured rate. An unavailable estimate is reported as unknown, never invented.
• Honest progress + safe abort. Progress lines read question 12/100 · ~1h 40m left (rolling ETA from measured per-item times); raw item ids appear only with --verbose (they always ride the --json payload). You can stop anytime by pressing Control+C — nothing is uploaded until the run finishes: SIGINT aborts cleanly (exit 130) and uploads nothing — a partial suite never scores, so it is never submitted, and a started suite is never time-truncated.

code-v1 runs the identical path: the runner administers the pooled code prompts and uploads OUTPUTS + timings, and the held-out tests never leave the server (they are the answer key — see executor/README.md). Because code-v1 is scored asynchronously (queue → ephemeral container → judge) rather than inline, the runner polls it to a final state on a longer per-benchmark window (15 min vs. core-v1's 5). The poll reports the SCORING fact and nothing else (0.1.8): a score, or parked (the server declined to score that submission — terminal, said plainly), or still working, in which case it keeps waiting. A run's status is the trust tier — unverified is what every new contributor's run is called — and is never read as a scoring verdict. A window that closes on a safely-landed run says exactly that, calmly, with the link; a run never confirmed as landed at all is the one case that still tells you to go and check. code-v1 uses the same sampler defaults as core-v1 (temperature 0, fixed seed): no in-repo code-v1 spec prescribes a different profile, and the runner does not invent one. Consent is shown once per consent version (state in ~/.crowdbench; the v2 receipt — 0.1.7 — adds the crowdbench.ai/terms + /privacy URLs and the contribution-grant line, so everyone sees it once more). The confirm screens carry a one-line pointer to the terms, and --agree-tos attests a human has seen and agreed to consent notice v2 (what uploads; the grant to Hooch Labs LLC; the public CC-BY-NC-SA 4.0 dataset; immutability; crowdbench.ai/terms) — required to upload non-interactively. After a run, the wrap-up ("your model qualifies for these industry suites too") prints only when at least one upload succeeded — an aborted session ends with its abort line and nothing else.

Commands

command	what it does
(no arguments, on a TTY)	the guided wizard — zero-file, zero-flag contribution flow
<benchmark-id>	run one benchmark end-to-end (the flag path); core-v1 / core-v1-rc / code-v1 / throughput-std-v1 in this phase
bridge lmeval:<task>:<variant>	run/parse an allowlisted lm-eval suite → validate → upload
detect	scan common local serving ports; report apps + loaded models (by route signature)
inventory	list local models (Ollama / LM Studio / HF cache / user dirs) + benchmark tooling
doctor	probe + connectivity + state report for support
cache list|clear	manage the content-addressed dataset cache

Key flags (dual-surface parity — every wizard prompt has a flag): --endpoint, --api-base, --dataset (omit to auto-fetch), --full (answer the complete pool instead of the server-issued subset), --model, --model-path, --reps N, --seed, --reasoning-effort none|minimal|low|medium|high, --reasoning-budget, --agree-tos, --minutes, --hf-repo/--hf-revision/--unattributed, --endpoint-api-key (env CROWDBENCH_ENDPOINT_API_KEY; never logged/uploaded/stored), --dry-run, --no-upload/--save/--upload-only, --json, --yes, --verbose (raw item ids on the progress lines), --sudo-probes, --ports. Auth uses CROWDBENCH_API_KEY (else an anonymous submitter is minted and cached).

Trust posture

• Outputs only. The runner uploads model outputs + telemetry; it never sees, computes, or transmits answer keys. Benchmark item content is inert data — never a tool, never agentic.
• Dataset integrity. A dataset's sha256 must match the benchmark's runner_spec pin before every run (capture requirement 11); a mismatch refuses to run (a tampered set never runs).
• Vendored artifacts, byte-matched. crowdbench_run/_artifacts/*.json are copied verbatim from packages/shared/artifacts so both languages read one source; CI fails on drift (scripts/check_artifact_drift.py — the third leg of the cross-language drift check).
• Endpoint key privacy. --endpoint-api-key is forwarded as the target endpoint's Authorization header only, never logged/uploaded/stored.
• Three network targets, exactly. The CrowdBench API, your inference endpoint, and — only when the identity step needs it and a human is present — one GET huggingface.co/api/models query carrying nothing but the served model name (owner-approved, 0.1.7; automatic and exact-match-only since 0.1.8, --no-hf-search to skip; fail-soft, never auto-picked, never on non-interactive runs).

Supply-chain audit trail: packages/runner/BUILD-TIME-CHECKS.md.

Publishing (PyPI Trusted Publishing)

Releases go out through .github/workflows/publish.yml using PyPI Trusted Publishing (OIDC) — there is no long-lived PyPI API token in this repo, ever. GitHub mints a short-lived OIDC token per run and pypa/gh-action-pypi-publish (pinned by full commit SHA) exchanges it with PyPI. Three human gates stand between a dispatch and a live release: workflow_dispatch only (no push/tag/schedule), a typed confirm input that must equal publish-to-pypi, and environment: pypi (owner approval before any publish step runs). A preceding job builds the sdist+wheel, runs twine check --strict, install-smokes both artifacts, and refuses the placeholder 0.0.x version line.

Owner one-time setup — verify on pypi.org. Create the Trusted Publisher for this project under Manage project → Publishing (or Your projects → Publishing for a pending publisher). These three fields must match the workflow exactly, or PyPI rejects the OIDC exchange:

PyPI Trusted-Publisher field	Must be
PyPI project (package) name	crowdbench-run
GitHub repository + workflow filename	crowdbench-ai/crowdbench-dev, workflow publish.yml
Environment name	pypi

(The owner also configures the pypi GitHub Environment's protection rule — required reviewers — so gate #3 actually pauses for approval.) This PR only prepares the workflow and docs; wiring the Trusted Publisher and the environment protection rule is an owner action on pypi.org / GitHub, and nothing publishes during the dark period.

Development

cd packages/runner
python -m pip install -e ".[dev]"
python -m pytest -q

The test suite runs on ubuntu/macos/windows in CI (a release gate — cross-platform posture).

---

CrowdBench is a service of Hooch Labs LLC. © 2026 Hooch Labs LLC · Apache-2.0.