OpenBench AI (v0.4.0)

A modular, open-source LLM experiment execution, evaluation, statistical comparison, and benchmarking platform.

Project Overview

OpenBench AI is a Python toolkit for running reproducible language model benchmarks. It evaluates model responses against standardized JSONL datasets, repeats experiments, compares models with appropriate statistics, collects blinded human preferences, and audits dataset and run quality. The core installation has no external runtime dependencies, and nothing requires a paid API.

Note: OpenBench AI is maintained by its project owner, and its initial development was AI-assisted. Contributions, bug reports, and new datasets are welcome!

What's new in v0.4.0

• Repeated-run benchmarking: openbench run --repetitions N --seed S creates a repeat group. Each repetition is its own run with recorded lineage: group ID, repetition index, seeds and seed status, the exact record IDs, per-record content hashes, and cache provenance.

• Interrupted groups resume without re-requesting completed items (openbench runs resume grp_...).

• Fix: repetitions no longer reuse cached responses. In v0.3.1, repeated runs reused cached responses across repetitions, which understated variability.

• Paired model comparison: openbench analyze aligns stored runs, repeat groups or evaluation files on stable record IDs, with no provider calls. It reports the following, with Holm (or BH) correction, an explicit missing-data policy, coverage and failure-rate comparisons, and category and difficulty breakdowns:

  • percentile bootstrap confidence intervals;
  • an exact McNemar test (binary metrics) or a paired sign-flip permutation test;
  • effect sizes;
  • wins, losses and ties;
  • optional equivalence testing (TOST) with a pre-specified margin.

• Compatibility checks: incompatible inputs, such as different record content, metric versions or scoring options, are refused unless you request a clearly labelled exploratory comparison.

• Repeated-run variability: generation variability (within items), item-to-item variation and provider failures are reported separately.

• Human preference evaluation: openbench preferences create|validate|analyze is a local-first, blinded A/B workflow. The task file contains no model identities; the identity key is a separate file. Analysis covers validated annotation import, task-weighted win rates with task-clustered CIs, tie and neither outcomes, position bias, Cohen's and Fleiss' kappa, and optional Elo with stated limitations.

• Quality audits: openbench validate-dataset PATH --quality-checks and openbench runs verify ID --quality-checks run read-only checks for:

  • duplicate IDs, and exact and near-duplicate prompts (a lexical 5-gram heuristic);
  • malformed or placeholder references, and keyword expectations missing from references;
  • invalid metric configurations and inconsistent versions;
  • missing outputs, incomplete runs and groups, integrity failures, and insufficient data.

  Audits never modify data.

• Reports: every new analysis renders to JSON, Markdown and CSV, with provenance (software, environment, input SHA-256s, seeds, parameters). CSV cells are protected against spreadsheet formula injection.

• Opt-in evaluation CIs: openbench evaluate --confidence-intervals. Output is unchanged without the flag.

v0.3.1 commands, flags, public Python names, the 200 dataset records and existing scores are preserved, and v0.3.1 run directories can be read without modification. See the migration guide for the details and the few documented behaviour changes.

Installation

git clone https://github.com/lakshaychhaba0001-sys/openbench-ai.git
cd openbench-ai

# Core installation (zero external runtime dependencies)
python -m pip install -e .

# Development + YAML config + JSON Schema validation
python -m pip install -e ".[dev]"

# Optional integration groups
python -m pip install -e ".[huggingface]"  # transformers + torch for local HF models
python -m pip install -e ".[semantic]"     # sentence-transformers for neural embeddings
python -m pip install -e ".[all]"          # All optional runtime integrations

# Statistical reference cross-checks (tests only; never imported by the library)
python -m pip install -e ".[dev,stats-ref]"

The Ollama adapter needs no extra packages: it uses the Python standard library HTTP client. All v0.4.0 statistics, preference and audit features use the standard library only.

Quick Start

1. Evaluate response files

openbench evaluate --dataset datasets/reasoning/sample.jsonl --responses examples/sample_responses.json \
  --metrics exact_match accuracy keyword_coverage --output results/

# With percentile bootstrap confidence intervals over items (v0.4.0, opt-in)
openbench evaluate --dataset datasets/reasoning/sample.jsonl --responses examples/sample_responses.json \
  --confidence-intervals --seed 0 --output results/ci/

2. Run models, with repetitions

openbench providers list
openbench run --dataset datasets/reasoning/sample.jsonl --provider mock --model model-a --repetitions 3 --seed 100
openbench run --dataset datasets/reasoning/sample.jsonl --provider mock --model model-b

openbench runs list
openbench runs show grp_<id>           # a repeat group
openbench runs verify grp_<id> --quality-checks
openbench runs resume grp_<id>         # finish an interrupted group

3. Compare stored results (no provider calls)

openbench analyze --runs grp_<id> run_<id> --baseline run_<id> --format markdown --output results/comparison.md
openbench analyze --results results/model_a.json results/model_b.json --equivalence-margin 0.05 --output results/cmp.json
openbench analyze --runs grp_<id> --output results/variability.json   # one repeat group: variability report

4. Collect blinded human preferences

openbench preferences create --runs run_<a> run_<b> --seed 7 --output tasks/ --key-output private/tasks.key.json
# annotators fill annotations.jsonl from tasks/tasks.jsonl
openbench preferences validate --key private/tasks.key.json --annotations annotations.jsonl
openbench preferences analyze --key private/tasks.key.json --annotations annotations.jsonl --format markdown --output results/preferences.md

5. Audit quality

openbench validate-dataset datasets/ --quality-checks --format markdown --output results/dataset_audit.md

A complete runnable walkthrough using the mock provider is in examples/v040_workflow.py.

CLI Commands

See the CLI reference for every flag.

Built-in Dataset Catalog (200 Examples)

The built-in quality audit reports 0 errors, 6 warnings and 8 informational findings. These are documented historical keyword exceptions, one ambiguous item and accepted answer variants; see quality audits. With 20 items per category, statistical comparisons on one category have low power, and the reports say so.

Metrics Reference

The kind (score_kind) selects the statistical procedure in analyze. Metric scores and versions are unchanged in v0.4.0.

Statistical honesty

• A non-significant difference is never reported as "no difference" or "equivalent". Equivalence requires a pre-specified margin and a TOST, and continuous-metric equivalence is disabled in v0.4.0 (descriptive only).
• Exact binary equivalence p-values are computed in floating point with a documented, tested error allowance. This is an engineering bound, not a machine-checked proof.
• Confidence intervals describe uncertainty over items for fixed models, prompts and settings. They are labelled as not adjusted for multiple comparisons.
• A seed passed to a provider does not guarantee deterministic output.
• SHA-256 checksums show that files match recorded checksums. They do not prove that results are correct or authentic.

See the statistical methodology.

Documentation Index

• v0.4.0 guides: Repeated runs | Model comparison | Human preferences | Quality audits | Reproducibility & reports | Statistical methodology | CLI reference | Python API | Migration from v0.3.1
• Execution, providers & integrity: Model Execution Engine | Providers | Ollama | Hugging Face (experimental) | Custom providers | Experiment tracking | Performance monitoring | Reliability & caching | Benchmark integrity | Security & privacy
• Datasets, metrics & configuration: Installation | Usage | Architecture | Dataset catalog | Core metrics | Advanced metrics | YAML profiles | Historical tracking | v0.3 compatibility
• Project: Changelog | Contributing | Security policy | v0.3.1 compatibility report

License

Released under the MIT License.