local-bench-ai

CLI benchmark runner for local-bench.ai — a community quality leaderboard for local AI setups. It benchmarks GGUF models served by llama.cpp (or any OpenAI-compatible endpoint), scores them on the five-axis Local Intelligence Index (Agentic, Knowledge, Instruction-Following, execution-verified Coding, Math — with call-formatting and long-context tracked as unweighted diagnostics), and packs signed, reproducible result bundles that publish to the board immediately on submission.

Quickstart

pip install "local-bench-ai[hf]"   # Python 3.11+

# 1. Fetch the complete benchmark suite (hash-verified)
localbench fetch-suite --site https://local-bench.ai \
  --suite suite-v1-full-exec-6axis-v1 --accept-suite-terms

# 2. Optional pre-cache (required with --offline; online advanced bench auto-caches a miss)
localbench cache-tokenizer <hf-model-id>

# 3. Run the full suite; explicitly consent to restricted model-generated code execution
localbench bench <catalog-model-or-hf-repo> \
  --llama-server-path <path-to-llama-server> \
  --allow-untrusted-code

# 4. Advanced managed-harness path
localbench bench \
  --runtime llama.cpp --server-bin <path-to-llama-server> \
  --model-file <model.gguf> --model-id <model-slug> \
  --hf-model-id <hf-model-id> \
  --suite suite-v1-full-exec-6axis-v1 --bench all \
  --wsl-venv-python <managed-wsl-python> \
  --appworld-root <managed-appworld-root> \
  --lane bounded-final-v2 --profile auto --tier standard \
  --allow-untrusted-code \
  --ctx 65536 --seed 1234 --out runs/my-bench

# 5. Submit — complete runs publish to the board immediately, attributed to you
localbench submit run --run runs/my-bench

Full-suite execution requires the AppWorld harness (localbench setup-agentic) and Docker. Agentic runs use the same signed, pinned appliance on both supported host paths: Windows hosts run it through managed WSL2, while Linux hosts materialize it natively and launch it under mandatory bubblewrap isolation. The other axes run wherever llama.cpp and Docker do. Runtime attestations preserve the shared canonical identity fields on both paths. Worker topology evidence is conditional: Windows/WSL emits wsl_distro, wsl_kernel, and appworld_root_under_mnt; native Linux emits runtime_topology, linux_kernel, and linux_os_release and omits those WSL-only fields. --allow-untrusted-code acknowledges the warning that model-generated code executes in a restricted container. Before model download, the CLI actively verifies its non-root, network-disabled, read-only, capability-free, seccomp-filtered, resource-bounded sandbox; missing consent or an unenforceable control fails the coding axis closed. Existing result bundles with pending coding artifacts can be completed with localbench grade-coding --allow-untrusted-code. Safetensors/vLLM execution is a separate maintainer-operated lane documented in docs/benchmark-build/vllm-maintainer-runbook.md; it does not change the public llama.cpp/GGUF path.

Troubleshooting

Windows CLI with a Docker engine inside WSL2

Do not use tcp://localhost:2375: the WSL2 localhost relay can drop Docker attach output even when ordinary daemon requests succeed. Connect through the current WSL adapter IP, pull the pinned image into the same rootful daemon store, and keep the distribution alive for the run. The complete setup, including rootless-vs-rootful stores, safe TCP exposure, transient systemd units, and a standalone version-matched Windows client, is in the Windows + WSL-engine coding sandbox guide.

For generic_think_tags_32768_v1 and gemma4_channel_32768_v1, use a 65536-token server context: their shared static budget is 32768 thinking + 16384 final tokens (49152 promised generated tokens). The old 8k profiles remain explicit historical operating points and are not compute-matched with this profile; the runner fails closed rather than silently lowering its context or budgets. Before its first scored run, the new Gemma profile must be validated end-to-end against the real gemma-4-31b-it on the RTX 5090.

The site's submit page generates these commands for your exact model and runtime, including the full identity flag set for bring-your-own-server runs. Publishable generic_think_tags_32768_v1 runs require a 65536-token server context.

What makes rows trustworthy

• Suites are hash-pinned releases; sampler settings are pinned (greedy, seeded).
• Coding is BigCodeBench-Hard, executed locally in a network-disabled, digest-pinned Docker sandbox with no host mounts; coding and agentic verdicts are carried as client-reported evidence and labeled as such on the board.
• Every number on the board links to a receipt with the full run manifest.
• Complete runs publish and rank immediately, attributed to the submitter; maintainers moderate post-hoc and can suppress rows that fail scrutiny.

Methodology: https://local-bench.ai/methodology