ModelGauntlet

Run every model through the same gauntlet.

ModelGauntlet is an open-source benchmark runner for comparing commercial, self-hosted, and routed language models through one reproducible workflow. It connects model providers to local diagnostics, maintained external evaluations, agent environments, and serving benchmarks, then writes normalized results and auditable reports.

Install the latest release from PyPI:

pip install modelgauntlet

It is designed for apples-to-apples comparison without pretending that every benchmark measures the same thing:

• Capability — reasoning, coding, instruction following, long context, structured output, and tool use.
• Agent system — an agent harness plus model, evaluated by outcome-verified tasks.
• Deployment — reliability, latency, throughput, and serving behavior.

ModelGauntlet is an orchestration and comparison layer. It does not reimplement or vendor the datasets and scorers maintained by upstream benchmark projects.

Contents

• Why ModelGauntlet
• Installation
• Quick start
• Model configuration
• Benchmark suites
• Scoring and comparison
• Judging
• Agent skill
• Outputs
• Sample result
• Optional integrations
• Private evaluations
• Adding benchmarks
• Development
• Security
• Third-party notices
• License

Why ModelGauntlet

Many evaluation tools focus on one model API, one benchmark family, or one metric. ModelGauntlet provides a common run manifest and result schema across those boundaries:

• Configure hosted APIs, OpenAI-compatible servers, Ollama, Open WebUI, or custom HTTP endpoints as model records.
• Run the same suite against one model, many models, or a historical collection of runs.
• Keep objective grading, model judging, agent verification, and serving telemetry distinct.
• Record benchmark releases, harness versions, parameters, prompt hashes, deployment fingerprints, and runtime fingerprints.
• Preserve raw artifacts alongside JSONL results and materialized Parquet and DuckDB data.
• Refuse or clearly label incomplete, incomparable, or environment-drifted comparisons.

The design follows practical conventions used by projects such as Inspect AI, Inspect Evals, lm-evaluation-harness, and SWE-bench: a small core, optional backends, explicit task configuration, reproducible outputs, and upstream-owned benchmark implementations.

Installation

ModelGauntlet requires Python 3.11 or newer. For normal use, install the published package:

pip install modelgauntlet

For checkout-based development, uv is recommended:

uv sync

The base installation includes the core runner and provider adapters. Optional integrations are installed separately:

uv sync --extra inspect    # Inspect AI and Inspect Evals
uv sync --extra harbor     # Harbor; Python 3.12+ and Docker required
uv sync --extra guidellm   # serving/load benchmarks
uv sync --extra litellm    # LiteLLM provider support
uv sync --extra dev        # tests, coverage, and Ruff

The pinned IFEval implementation has an upstream Git dependency that cannot be represented in PyPI package metadata. Install it separately only when the Inspect environment requires it:

uv pip install \
  "instruction-following-eval @ git+https://github.com/josejg/instruction_following_eval@0c495b2"

After installation, both modelgauntlet and the shorter mg command are available.

Quick start

Create a local configuration from the example:

cp configs/models.example.yaml configs/models.yaml

Set the required environment variables, then validate and probe the models:

export OPENAI_API_KEY=...
export OPENAI_MODEL=...

uv run modelgauntlet models validate
uv run modelgauntlet models probe

Run the diagnostic suite:

uv run modelgauntlet run \
  --models local-openai-compatible \
  --suite core

Generate reports using the returned run ID:

uv run modelgauntlet report <run-id> \
  --formats html,markdown,json,csv
uv run modelgauntlet analyze <run-id> --profile capability

The core, comprehensive, and stress suites are diagnostic regression suites. They are useful for checking prompts, providers, retries, timeouts, and local behavior; they should not be presented as broad public model-quality claims.

For maintained objective benchmarks:

uv sync --extra inspect
uv run modelgauntlet run \
  --models candidate \
  --suite objective-smoke

Use objective for the full configured objective catalog. Full external runs can be expensive and may require additional upstream dependencies or explicit privileged-container consent.

Model configuration

Models are declared in YAML. API credentials are referenced by environment variable name and should never be written directly into configuration:

models:
  - id: local-vllm
    protocol: openai_compatible
    base_url: http://localhost:8000/v1
    model: served-model
    capabilities:
      chat: true
      tools: true
      streaming: true
      usage: true
    deployment:
      engine: vllm
      hardware: unknown

  - id: hosted-model
    protocol: openai
    model: ${OPENAI_MODEL}
    api_key_env: OPENAI_API_KEY

Supported protocols include:

Protocol	Typical use
openai	OpenAI hosted API
anthropic	Anthropic Messages API
openai_compatible	vLLM, SGLang, TGI, local gateways, and proxies
ollama	Ollama server
openwebui	Open WebUI gateway
custom_http	Templated HTTP request/response mapping
litellm	Optional LiteLLM routing

Give separately routed endpoints separate model IDs. This keeps proxy and deployment behavior visible in reports instead of hiding it behind one name. See configs/models.example.yaml and gauntlet.yaml.

Benchmark suites

Suites live in suites/ and specify the engine, benchmark releases, comparison mode, repetitions, concurrency, retries, timeouts, and score weights.

Suite	Purpose	Engine
smoke	Small local diagnostic run	core
core	Local diagnostic regression suite	core
comprehensive	Larger local diagnostic coverage	core
stress	Local concurrency and timeout testing	core
objective-smoke	Small external capability run	Inspect
objective	Full configured capability catalog	Inspect
livebench	Pinned LiveBench environment	Inspect
agent-system	Outcome-verified terminal agents	Harbor
serving	Load and serving behavior	GuideLLM

The external catalog in benchmarks/catalog.yaml records the upstream task, release, harness version, profile, metric contract, source, and license pointer. Validate suites before running them:

uv run modelgauntlet suites validate objective-smoke
uv run modelgauntlet suites validate objective

Recommended evaluation workflow

Do not run the largest possible catalog for every model change. Use staged evaluation:

1. Smoke: run a small local suite to verify connectivity, configuration, grading, and basic regressions.
2. Targeted: select a small external subset while developing. Use the benchmark that matches the capability you are changing—for example, BFCL for tool use, IFEval for instruction following, or InfiniteBench for long-context behavior.
3. Release evaluation: run the broader external catalog when you need a durable model comparison or release report. These runs can be slow and expensive, especially long-context and agent/container benchmarks.
4. Serving evaluation: run serving separately when measuring throughput, latency, or GPU utilization. Do not mix serving load with capability claims.

External selections support a configurable item limit for practical diagnostic runs. A limited run is useful for validation and iteration, but its scores have low statistical resolution and should not be presented as definitive model quality claims. Full suites are best reserved for scheduled evaluations, benchmark cards, or research comparisons. See docs/benchmark-authoring.md for selection and release-pinning guidance.

Scoring and comparison

Every run produces a normalized item-level record. Reports expose three scorecards rather than one opaque leaderboard score:

• Capability uses graded benchmark items and objective or subjective quality.
• Agent-system uses stable agent-plus-model subjects and verifier rewards.
• Deployment uses deployment records for serving reliability, latency, and throughput.

Legacy dimensions such as quality, reliability, efficiency, serving performance, and agent effectiveness remain available for compatibility. Missing telemetry is null, not zero. Missing weighted dimensions do not silently improve a headline score.

Run comparisons require compatible benchmark releases, harness versions, parameters, prompt hashes, score weights, and strict/native modes. Historical comparisons recompute peer-relative efficiency and serving scores across the selected runs:

uv run modelgauntlet compare <older-run-id> <newer-run-id>

Runtime and deployment fingerprints prevent latency comparisons across changed client or serving environments. To compare quality while suppressing serving scores:

uv run modelgauntlet compare <run-a> <run-b> \
  --allow-environment-drift

See docs/scoring.md for the scoring model and references/report-interpretation.md for how to read reports.

Judging

Prefer deterministic assertions whenever possible. Local packs support exact, containment, regex, numeric, JSON, and tool-call assertions. Rubrics and pairwise judgments are available for work that cannot be graded reliably by deterministic checks.

Use a configured model as an LLM judge:

uv run modelgauntlet judge <run-id> --judge local-judge

The active Cursor, Codex, or Claude Code harness can explicitly judge blinded tasks:

uv run modelgauntlet judge <run-id> --judge agent:self
# inspect only agent-judge-tasks.jsonl, then write one JSONL answer per task
uv run modelgauntlet judge-import <run-id> \
  --responses judgments.jsonl \
  --harness cursor

agent:self is never selected automatically. Its identity is recorded as an agent harness and self-judge conflicts require explicit acknowledgement. See references/judging-policy.md.

Agent skill

SKILL.md provides the workflow for an AI coding agent operating ModelGauntlet. It covers model validation, suite selection, preflight cost and runtime warnings, external benchmark safeguards, profile-specific analysis, historical comparisons, human-readable reports, and explicit agent:self judging.

The skill includes representative evaluation prompts in evals/evals.json so its workflow can be checked against provider comparison, full capability evaluation, datacenter GPU serving, and self-judging requests.

Outputs

Each run is stored under results/<run-id>/:

run.json              # manifest, fingerprints, configuration, and status
results.jsonl         # canonical item-level result records
results.parquet       # columnar materialization
run.duckdb             # queryable result and manifest metadata
raw-artifacts/        # provider responses and upstream artifacts
reports/              # generated HTML, Markdown, JSON, and CSV reports

External harness runs are immutable audit units. They cannot resume in place; rerun --all creates a new run. Partial external runs return a nonzero exit status unless --allow-partial is explicit, and remain non-publishable:

uv run modelgauntlet rerun <external-run-id> --all
uv run modelgauntlet run --models candidate --suite objective-smoke \
  --allow-partial

Sample result

Results are designed to be readable by people first and queryable by tools second. The report starts with an executive summary, then separates capability, agent-system, and deployment scorecards. It also explains missing coverage, confidence intervals, cost, latency, and reproducibility fingerprints.

Example human-readable report:

# ModelGauntlet report: sample-run

## Executive summary

Capability quality: 82.4 / 100
Capability coverage: 100.0%
Reliability: 98.7%
Median latency: 420 ms
P95 latency: 680 ms
Output throughput: 126.5 tokens/sec
Estimated cost: $3.42
Quality 95% CI: 79.8–84.6

Capability: 82.4 (publishable)
Agent system: n/a (not measured)
Deployment: 88.6 (publishable)

The complete synthetic examples are in examples/sample-results/:

• executive-summary.md is the human-readable version.
• comparison.json is the machine-readable scorecard.

These examples are illustrative and are not results from a real model.

Optional integrations

The integrations are boundaries around upstream tools, not core dependencies:

• Inspect AI and Inspect Evals provide maintained capability benchmarks.
• Harbor provides containerized, outcome-verified agent tasks.
• GuideLLM provides serving profiles and load measurements.
• NVIDIA DCGM provides optional GPU telemetry for local serving runs.

modelgauntlet doctor reports which optional executables are installed:

uv run modelgauntlet doctor

LiveBench intentionally uses an isolated upstream environment because its dependencies can conflict with the shared Inspect installation. Set MODELGAUNTLET_LIVEBENCH_INSPECT to the pinned environment's inspect executable as described in suites/livebench.yaml.

Private evaluations

Private rotating evaluations stay outside this repository. Point MODELGAUNTLET_PRIVATE_CATALOGS at one or more catalog YAML files:

export MODELGAUNTLET_PRIVATE_CATALOGS=/secure/path/private-catalog.yaml

Mark entries visibility: private and public_item_ids: false. ModelGauntlet hashes item identifiers, avoids packaging private task content, and emits metadata-only benchmark cards.

Adding benchmarks

For local diagnostic packs, add a YAML file under benchmarks/prompts/ with:

• stable test IDs;
• category and difficulty;
• explicit point values;
• deterministic assertions or a rubric;
• license metadata in benchmarks/LICENSES.yaml.

For an external evaluation, add a catalog entry with an exact release, adapter, upstream task, harness version, primary metric, declared range, source URL, and aggregation policy. ModelGauntlet uses a versioned adapter envelope containing the adapter identity, protocol version, expected item count, observed item count, artifacts, and normalized records.

See docs/benchmark-authoring.md.

Development

uv sync --extra dev
uv run ruff check .
uv run pytest --cov=modelgauntlet --cov-report=term-missing \
  --cov-fail-under=80
uv build

Contributions should preserve result-schema compatibility, include tests for new behavior, and never commit credentials, private prompts, model outputs, or generated result directories. See CONTRIBUTING.md.

Security

Model outputs and provider responses are untrusted. The core runner does not execute generated code. Use the isolated Inspect or Harbor environments for execution-scored coding and agent tasks. Never put credentials in model YAML, request templates, benchmark prompts, or committed artifacts.

Report vulnerabilities privately through the repository's security advisory feature. See SECURITY.md.

Third-party notices

See THIRD_PARTY_NOTICES.md for the projects, benchmarks, dependencies, and GPU tools ModelGauntlet integrates with or references, including attribution and redistribution boundaries.

License

ModelGauntlet is released under the MIT License. See THIRD_PARTY_NOTICES.md for upstream benchmark, dependency, engine, and GPU-tool attribution and redistribution boundaries.