CrowdBench

CrowdBench measures local language models and contributes intelligence, speed and memory results to https://crowdbench.ai. The open-source runner is being prepared for the crowdbench 1.0 package. It is currently published as crowdbench-run; the package and command rename to crowdbench is coming.

Install and run CrowdBench with uv:

uvx --python 3.14 crowdbench-run

The guided wizard finds local engines, helps select a model and measurements, and asks for the required agreements before model work and uploads. The runner requires Python 3.14 or later; uv can obtain it. Use uvx --python 3.14 crowdbench-run --help for commands, uvx --python 3.14 crowdbench-run detect for serving engines and models, and uvx --python 3.14 crowdbench-run run --help for explicit benchmark options.

Linux, macOS and Windows are supported, with suite-specific limits. Native Windows does not support lm-eval code-execution suites, EvalPlus, or stock math scorers requiring signal.SIGALRM. WSL2 supports intelligence measurements, but public speed and memory measurements are unavailable. HTTP adapters include Ollama, llama.cpp, ik_llama.cpp, LM Studio, mlx-lm, oMLX, vLLM, LocalAI and Strata; suite capabilities and model evidence are still required. Isolated CrowdBench Engine runtimes use local MLX weights on Apple-silicon macOS or GGUF weights on compatible hosts with supported native wheels. Generative suites in the GGUF runtime are currently refused, including local-only runs.

Before running a model, the runner checks that you have accepted the current Terms of Service, even for work kept local. Uploading also requires agreement to the current Contribution Terms. One prompt can cover both agreements; --agree-tos records a human's acceptance in scripted use. Checks that execute generated code ask for another permission, supplied with --allow-code-exec on each scripted run. lm-eval's evaluator does not isolate that code from your machine.

Contributions can send scores, speed readings, harness result files, and full validation answers with reasoning and item timings. Full public-suite responses and reasoning are uploaded for private rescoring and audits rather than public display. Uploads also describe hardware, operating systems and run conditions, machine ownership or rental status and provider, engine configuration, setup answers and engine errors, model names, file metadata and fingerprints, the runner version and the recorded legal and code-execution acceptances. Session plans, skips and refusals with reason codes, and answer-length totals are sent too. During a check, periodic messages identify your contributor key and the active check, with request counts when available; they contain neither answers nor unfinished results.

The upload filters aim to omit direct identifiers such as your name, email, hostname, IP address and device serials. They also shorten recognized paths and mask recognized credentials, but filtering is incomplete: model responses, file metadata, copied names and engine errors can still expose personal information or secrets. A public contributor ID connects submissions made with the same key; the absence of an account name does not make those submissions anonymous.

This is an overview. Read the Privacy Policy for the exact privacy text, and run crowdbench-run --explain terms for the exact bundled agreements.

The runner fingerprints model weights and records configuration, harness and dataset evidence; missing observations stay unknown. Industry suites use their named harness and scoring rules. lm-eval's GSM8K extractor can score a correct answer wrong, and its MBPP+ suite checks the single shown assertion rather than EvalPlus's extended tests. Server checks remain separate; an upload receipt is not final acceptance. Submitted measurements can contain errors or deliberate manipulation; these checks do not independently verify every result. The source distribution includes the public README and design guides explaining these limits.

To contribute code, run uv run --extra dev pytest from the runner source directory. Send security reports through SECURITY.md.

Apache-2.0 · © 2026 Hooch Labs LLC