calm-benchmark 0.1.5

An open, calibrated benchmark for menopausal-status prediction from routine health markers

CALM

Calibrated Menopausal-status benchmark — an open, reproducible benchmark for predicting menopausal status from routine, low-cost health markers.

0.1.5 · Python 3.14 · uv · MIT

CALM provides a rigorously specified benchmark for: premenopausal vs. postmenopausal from routine markers (labs, body measures, demographics), with calibrated confidence, an explicit no-call option, honest generalization reporting, and end-to-end provenance. It is a benchmark engine — a library and CLI — not an application. It emits versioned, auditable artifacts that downstream tools consume.

Out-of-training-range inputs are always withheld. This is documented univariate range checking rather than a claim of multivariate novelty detection.

Reported metrics include deterministic bootstrap confidence intervals. Synthetic runs are explicitly exploratory; the default real-data protocol is within-cycle frozen-test evaluation, not external validation.

---

Install

Requires Python 3.14 and uv.

After the first PyPI release:

uv tool install calm-benchmark
calm-benchmark --help

For development from source:

git clone <repo> && cd calm
uv sync
uv run calm-benchmark --help

---

Quickstart

# run the deterministic, offline benchmark demonstration
uv run calm-benchmark demo --output runs

The demo writes, under runs/synthetic-demo/, metrics.json, a reliability plot, a benchmark card, and a complete manifest.json.

Pinned public NHANES run

The public 2017–2018 run consumes the official XPT files already downloaded to data/raw/, verifies their SHA-256 hashes, validates the cycle schema, and records source URLs and checksums in its manifest.

uv run calm-benchmark run-nhanes --config configs/default.toml --run-id nhanes-2017-2018

It uses the versioned public_cessation outcome: RHQ031 and RHD043 distinguish reported current periods from menopause/change-of-life cessation and exclude reported hysterectomy. Public data redact pregnancy and hysterectomy information for women aged 45 and older, so this run does not claim to exclude surgical menopause in that age band. An RDC-only natural-menopause variant is outside this public benchmark.

Attribution and optional explanations

Each benchmark run emits attributions.json: deterministic, per-prediction feature-association scores with model-hash provenance. They describe association under a fixed reference replacement, not cause. The optional OpenAI explanation service lives in services/explainer/; it receives only a versioned CALM hand-off and refuses withheld/OOD results, missing attributions, diagnosis, and treatment requests.

---

The benchmark contract

• Task — binary premenopausal vs. postmenopausal (optional secondary: early natural menopause).
• Split — deterministic, stratified, content-hashed; the hash is the no-leakage proof.
• Metrics — balanced accuracy, per-class recall, F1, AUROC, PR-AUC, Brier and reliability, no-call rate, bootstrap confidence intervals, and subgroup breakdowns with denominators and eligibility.

See DESIGN.md for the full specification and domain rationale.

---

Reproducibility & provenance

Every artifact carries a Provenance record (normalized-config hash, input hashes, tool and git versions). Re-running the same config against the same pinned inputs yields identical hashes. NHANES public-use data are U.S. public domain; the NCHS Data User Agreement notice is preserved in exports.

---

Architecture at a glance

src/calm/
├── config/         # pydantic schemas + loading
├── domain/         # entities, value objects, invariants, registries (pure — no I/O)
├── ports/          # Protocol interfaces
├── adapters/       # NHANES source, synthetic source, codebook parsing
├── pipeline/       # acquire ingest cohort labels features split train calibrate evaluate report
├── orchestration/  # instrumented runner, run manifest
├── observability/  # logging, timing, hashing, provenance
└── cli.py          # Typer app

Full details in DESIGN.md. The AI-agent build process is governed by AGENT.md; the roadmap lives in TODOS.md.

---

Extending CALM

• New cycle — add a CycleSpec in config.
• New label rule / outcome — implement and register a LabelRule.
• New data source — implement the DataSource port.
• New model or metric — register a factory.

---

Development

uv run ruff check && uv run ruff format --check
uv run mypy
uv run pytest --cov=calm
uv run pre-commit run -a
uv run mkdocs serve

Standards: full type hints, Google-style docstrings on the public API, no superfluous comments, unit tests required, property-based tests for pipeline invariants, and a coverage gate on the core.

---

Contributing

See AGENT.md for engineering conventions and TODOS.md for the roadmap. Pull requests must pass all gates and include tests and a CHANGELOG.md entry.

---

Versioning & changelog

Semantic Versioning; history in CHANGELOG.md (Keep a Changelog).

---

License & disclaimer

MIT. Research benchmark, not a medical device. CALM produces population-derived predictions for methodological research and must not be used for clinical or diagnostic decisions.

---

Citation

CALM: A Calibrated Benchmark for Menopausal-status Prediction from Routine Health Markers. <year>. <repo>.