model-gym 0.8.0

HELM-style multi-leaderboard AI benchmarking platform — modernization, healthcare, frontier safety, agentic, code patches, compliance

Model Gym

HELM-style, multi-leaderboard AI benchmarking for enterprise — legacy modernization, healthcare AI, frontier safety, agentic tool-use, code patch generation, and per-vertical / per-regulation evaluation, all under one FastAPI service with shared persistence, an HMAC-chained audit ledger, evidence-bundle export, and an Inspect-AI-compatible eval-log exporter.

40 leaderboards. 19+ metrics. 780+ tests green. Last sync: 2026-05-17 (Wave 9 Track A/B).

---

What you can do today

Source: docs/diagrams/request-flow.mmd — render via npx -y @mermaid-js/mermaid-cli -i docs/diagrams/request-flow.mmd -o request-flow.svg

flowchart LR
    User["User / CI / dashboard"] --> API["FastAPI<br/>app/main.py"]

    API --> Core["HELM-style<br/>eval-core<br/>app/eval/"]
    API --> Legacy["Legacy<br/>GymSpec runtime<br/>app/engine.py"]

    subgraph Boards["40 leaderboards (auto-discovered)"]
      Modern["Modernization × 5<br/>classic / evidence / robustness<br/>agentic v1.2 / safety"]
      SWE["SWE-Bench Verified<br/>(struct + Docker exec)"]
      Med["MedHELM (10 RunSpecs)"]
      Vert["Per-vertical × 17<br/>HCLS · retail · fin-svc<br/>hi-tech · industry · edu-K12"]
      Comp["Compliance × 9<br/>HIPAA · PCI-DSS · SOX · GDPR<br/>CMS · HTI-1 · 3 state AI<br/>OWASP × 2 · MITRE ATLAS"]
      Safe["Frontier safety × 2<br/>WMDP · medical_safety<br/>+ agent_threat_safety"]
    end

    Core --> Boards

    Boards --> Snap["Postgres :25432<br/>leaderboard_snapshots"]
    Boards --> Traces["traces/<run_id>/*.json"]
    Boards --> Ledger["audit_ledger.jsonl<br/>(HMAC chained)"]

    Snap --> Bundle["Evidence bundle ZIP<br/>/evidence_bundle"]
    Traces --> Bundle
    Ledger --> Bundle

    Snap --> Inspect["Inspect AI EvalLog v2<br/>/inspect_log"]
    Traces --> Inspect

• Run any of 30 leaderboards against any model with hash-keyed request caching
• Stream HuggingFace datasets with access_tier: public | gated | private (set HF_TOKEN for gated, MODELGYM_HF_OFFLINE=1 for airgap / CI)
• Score with 19+ metrics across 7 groups (core, evidence, HCLS, compliance, agentic, safety, code)
• Export an Inspect-AI-compatible EvalLog v2 JSON for any run
• Export an evidence bundle (snapshot + traces + audit slice + scenario fingerprints) per run
• Drive the legacy GymSpec onboarding/certification path on the same store

---

Leaderboard catalog

Category	ID prefix	Boards	Headline metric
Modernization (core)	modernization_*	5	varies per board
· classic	modernization_classic	1	exact_match + judge
· evidence	modernization_evidence	1	citation_f1 + hallucination_rate
· robustness	modernization_robustness	1	robustness_delta
· agentic	modernization_agentic v1.2	1	step_success_rate + action_set_jaccard + task_completed_partial (TAU-Bench live mocks)
· safety	modernization_safety	1	wmdp_safety (inverted)
SWE-Bench Verified	swe_bench_verified v1.1	1	patch_files_match_gold + swe_bench_resolved (opt-in Docker)
MedHELM	medhelm_modernization	1	jury_score + bertscore
Vertical	vertical_*	17	per-vertical jury_score
Compliance	compliance_*	9	control_match (HIPAA, PCI-DSS, SOX, GDPR, CMS-Interop, HTI-1, NYC LL 144, CA AI laws, IL AI laws + OWASP LLM Top 10, OWASP Agentic, MITRE ATLAS)
Frontier safety	modernization_safety, medical_safety	2	wmdp_safety, red_team_safety (inverted)
Agent threat	agent_threat_safety	1	atr_safety (ATR rule pattern coverage)

For the full registry: uv run python scripts/export_registry.py.

---

Quick start

# install (uv is the toolchain)
uv sync

# start Postgres on 25432 + apply migrations
docker compose up -d db
uv run alembic upgrade head

# run the full eval test suite (offline by default)
MODELGYM_HF_OFFLINE=1 uv run pytest tests/eval/ -q

# boot the API + dashboard
uv run uvicorn app.main:app --reload --port 8000
# → http://localhost:8000/leaderboards.html
# → http://localhost:8000/dashboard         (legacy GymSpec UI)

Optional live data

# pull real HF datasets (MedHELM, WMDP, SWE-Bench Verified, etc.)
unset MODELGYM_HF_OFFLINE
export HF_TOKEN=hf_xxx      # only needed for gated-tier scenarios
uv run pytest tests/eval/test_medhelm_real_data.py -q

---

Environment variables

Var	Purpose	Required in prod
DATABASE_URL	Postgres DSN (default: :25432/modelgym)	yes
AUDIT_HMAC_SECRET	HMAC key for audit ledger chain	yes
MODELGYM_AUDIT_LEDGER	Path to append-only ledger JSONL	yes
MODELGYM_TRACE_ROOT	Root dir for per-instance trace JSON	yes
MODELGYM_REQUEST_CACHE_PATH	SQLite path for hash-keyed model-call cache	yes
MODELGYM_HF_OFFLINE	1 to forbid HF network calls during tests/airgap	no
HF_TOKEN	HuggingFace token — required for access_tier=gated	when gated used

---

Adding a new benchmark

1. Drop a scenario class in app/eval/specs/<your_thing>.py (subclass HuggingFaceDatasetScenario for HF, or implement Scenario directly).
2. Drop a leaderboard registration in app/eval/leaderboards/<your_thing>.py (calls REGISTRY.register(Leaderboard(...)) at import time).
3. FastAPI auto-discovers it on next boot. Add a test under tests/eval/test_<your_thing>.py.

For a worked example, see app/eval/specs/wmdp_scenarios.py + app/eval/leaderboards/modernization_safety.py + tests/eval/test_wmdp.py.

---

Documentation map

• architecture.md — full architectural truth (read this first when extending)
• ROADMAP.md — what shipped, what's next
• docs/WAVE_LOG.md — Plans 1-7 vs ship reality + Waves 1-5
• docs/superpowers/plans/ — original written plans (superseded; WAVE_LOG is canonical)
• PATH_TO_PROD.md — known prod-readiness boundaries

---

Boundaries (not yet production-safe)

• X-Gym-Role header is dev-mode only. Set MODELGYM_ENV=production to fail-close — Keycloak is then required (or MODELGYM_ALLOW_DEV_AUTH=1 for airgap deploys that issue their own local JWTs).
• SWE-Bench Docker execution (FAIL_TO_PASS / PASS_TO_PASS) is opt-in via MODELGYM_SWE_BENCH_EXEC=1 + a docker daemon + the upstream swebench/sweb.eval.* per-instance images. Without it, scoring is structural-only.
• WMDP scoring remains structural (frontier-safety MCQ). TAU-Bench has live retail + airline mocks (Waves 7.4 + 7.5); 4 of 12 vendored tasks have gold_final_state for TaskCompletionMetric — remaining authoring is rolling content work.
• BERTScore falls back to rouge_L when the bert_score package isn't installed (set MODELGYM_BERTSCORE_REQUIRE_REAL=1 to fail-close).
• Inspect AI export emits both .json and a real in-tree .eval zipfile (Wave 7.1) — no inspect-ai dep required.

---

License

See LICENSE.md.