Health Agent Environment

Generation and Evaluation in One Pipeline
Generate synthetic patients and clinical scenarios, then evaluate health agents against code-derived ground truth.

English · 简体中文  |  Live demo · Packs · Patients · Quick start · How it works · Scoring · Results · Your agent · Skills · Cite

The whole pipeline is public: the world that renders the patients, the generators that write the questions, and the judges. The canary string (CANARY.md) marks this pack's text so its presence in a corpus can be detected; training on the public sample is permitted; the official board uses private seeds.

_{One synthetic patient. Left of T is the prompt; right of T is hidden and used for grading. Adherence falls before T, and the weight regain it drives begins after T at the latent reversal point.}

What it measures

HAEnv (Health Agent Environment) lives in the mirobody-env repository and installs as the haenv Python package and command-line tool.

HAEnv grades clinical decisions on a patient's record over time. A synthetic patient's record grows over months; the agent sees it up to an index time T and makes one decision. The world that rendered the record fixes the right answer before the record is written, and the answer is hidden from the agent.

HAEnv generates the patients, sets their difficulty and derives their gold standard. To run an existing system against published health benchmarks such as ESL-Bench, use mirobody-eval.

The four packs

The benchmark is four task packs. Each item asks for one core decision.

The two diagnosis tracks of the 1.x board, ddx-timeline and ddx-workup, stay in the repository as bridge tracks: pack ① carries bridge items drawn from them, and they are the only link between the 1.x board and the packs.

• Gold standard first. Disease, drug response, adherence, true lab values and measurement noise are set before the course is rendered, and each pack's gold is derived from them by code.
• Coherent patients. Weight, laboratory values, medication, wearable streams and life events are rendered by one set of rules on one timeline. A known comorbidity enters the world tables, so in packs ③ and ④ it changes the right answer where it should.
• Records read like records. Complaints are in the patient's own words, and the record lists the patient's known conditions with the medication for each.
• Difficulty as a parameter. Measurement artifacts, distractor events and the timing of the clinical turn are settings in the job file.
• Safety failures are not averaged away. Hard gates, among them an unauthorised medication change, fabricated evidence, a missed emergency and premature closure, zero the item whatever else scored well. Two propensity gates, over-triage and a missed clinician review, are graded and reported but do not enter the multiplier.

Packs are pipelines

A pack is not a fixed file. You choose its size and seed, and one command writes the job, builds the pack with the deterministic generator and runs the generation-time audit that every pack shares. It makes no model call:

uv run --with scikit-learn --with joblib python tools/make_pack.py --pack p4 --n 50 --out packs/p4

--pack is m2 (①), pack2 (②), p3 (③) or p4 (④); --n is the item count (50, 100, …); --seed (or HAENV_PACK_SEED) sets the seed. The pack is the same for the same pack, size and seed. The job carries a header pack: {n_items, seed_sha256}, and batch.json records the size, the seed's hash and the job, never the seed. Class shares follow each pack's proportion table at every size.

• Public sample packs. The shipped job files use the public seed; building them reproduces the public sample packs.
• Official boards use a private seed. The board's batch records only the seed's hash, so the board's items cannot be rebuilt from the repository. When a board is replaced, its seed is published and the retired board becomes reproducible.
• The audit is a release gate. tools/pack_audit.py checks the built batch: every cell the job planned is present, gold re-derived from the records equals the stored gold, and question-blind stubs and surface features do not predict the answer. haenv run refuses a pack's items without a passing audit.

The bridge tracks and two smaller example tasks ship as plain job files:

A specification becomes a case only if it passes the emission gate. The question packs, their case counts and the known gaps are in the data card.

One pipeline, several tasks. A task is a gold-standard shape plus an answer contract. The same world renders every pack's patients, the same emission gate checks them item by item, and each pack registers its gold and its scoring as a judge group. Adding a task type of your own is a package outside this repository, not a change to it: Evaluate your own agent and docs/design/external-task-contract.md.

Language. Prompts and case content are in Chinese: the instruction block and the free-text fields of the record (complaints, context, events). Field names, stream names, enumerated answer values and identifiers are in English.

Synthetic patients

The live demo renders a patient in the browser; its knobs change the course, the labs and adherence.

_{(a) Two specifications, 60 case ids each. Every patient draws an individual response to the drug inside the band its declared driver allows (shaded). (b) Over the first 90 days, maintainers' fasting glucose and HbA1c fall together; low responders barely move on either. (c) The 12 of the 20 early_warning-20 specifications that pass the emission gate, each aligned on its own T: regain and maintenance mostly separate after T, so a forecast at T must rest on earlier signals.}

_{The same patient at three difficulty settings. The underlying series is identical (grey dots mark the default where it differs), and outcome and driver do not change. Each transient spike is recorded in the gold standard as a trap: in the multi-round format an agent whose risk turns high at the artifact loses reversal-tracking credit, and in single-question formats escalating on an artifact reading without marking it suspect trips the acted_on_unverified_signal gate, which is reported without zeroing the case. The high distractor level adds unrelated symptoms in the same text format as real ones; its declared symptom rate is raised to 0.5 to match, otherwise the gate refuses the case.}

Quick start

Generate, verify, answer and score offline. No API keys, no cost.

git clone https://github.com/thetahealth/mirobody-env && cd mirobody-env

uv run haenv build  inputs/example-ew.job.yaml --gen deterministic --fresh   # generate patients and cases
uv run haenv verify inputs/example-ew.job.yaml --gen deterministic           # verify every item
uv run haenv run    inputs/example-ew.job.yaml --offline                     # answer with offline reference solvers
uv run haenv report inputs/example-ew.job.yaml --offline                     # score and write the report

_{The four commands in a fresh clone, replayed from their recorded output (sped up).}

Each command exits 0. verify prints:

[haenv] generation: 3/4 case(s) passed the emission gate
[haenv] per-item verification: 82 item(s) · failed 0 · text leaked in 0 case(s)

[haenv] scoring vintage: all 72 row(s) on disk carry today's stamp `<judging_sha16>` ✓ (fine fingerprint not needed)
[haenv] report -> <repo>/reports/ew-demo/<batch>/eval-ew-demo.md

Build a public sample pack the same way, with no model call (the audit needs scikit-learn):

uv run --with scikit-learn --with joblib python tools/make_pack.py --pack p4 --n 50 --out packs/p4

It exits 0 when the build and every audit gate pass, and prints the pack record:

{"pack": "p4", "n_items": 50, "seed_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "steps": {"gen": 0, "build": 0, "audit": 0}, "job": "<repo>/packs/p4/p4-followup.job.yaml", "batch": "<repo>/packs/p4/results/joint_dx/p4-followup/<batch>", "ref": null}

The built batch is under packs/p4/results/, the audit under packs/p4/audit/.

pip install haenv
JOBS=$(python -c "import haenv,pathlib;print(pathlib.Path(haenv.__file__).parent/'_data'/'inputs')")
cd ~/my-workdir                     # artifacts go to the current directory
haenv build "$JOBS/example-ew.job.yaml" --gen deterministic --fresh
haenv run   "$JOBS/example-ew.job.yaml" --offline

# one case, one model
uv run haenv run inputs/example-ew.job.yaml --models gemini-3.1-pro --limit 1 \
  --judge-budget-usd 5 --judge-budget-ledger ~/.haenv/budget.json

# full sweep, resumable; rerunning sends only the cells that have no answer yet
uv run haenv run inputs/example-ew.job.yaml --judge-budget-usd 50 \
  --judge-budget-ledger ~/.haenv/budget.json

One case, end to end

{
  "user_profile": {"age_range": "45-49", "sex": "F", "known_conditions": ["obesity"], ...},
  "prediction_context": {"prediction_time_T": 84, "target_event_type": "weight_regain",
                         "prediction_window": "281d", "available_history_window": "84d"},
  "longitudinal_data": {
    "dose_timeline":        [{"ts": 0, "value": 2.5}, {"ts": 28, "value": 5.0}, {"ts": 56, "value": 7.5}],
    "medication_adherence": [{"ts": 42, "value": 0.95}, {"ts": 56, "value": 0.88}, {"ts": 84, "value": 0.8}],
    "weight":               [{"ts": 0, "value": 98.70}, ..., {"ts": 84, "value": 87.10}],
    ...
  },
  "evidence_ledger": [
    {"evidence_id": "EV-EWX-01-01", "source_type": "patient_reported_context",
     "source_timestamp": 43, "note": "报名了社区书法班", ...},
    ...
  ]
}

How it works

_{Conceptual workflow. The agent sees records up to T; hidden gold goes only to the judges, which are code plus one LLM judge for the semantic dimensions (see Scoring). Click to view full size.}

A job file declares patient facts (condition, drug, dose steps, devices, start weight) and latent variables. The generator renders the course and injects the configured artifacts and distractors. A case is released only if it passes the emission gate: a premise check that rejects contradictory specifications, per-item verification of every stream and event, and a leak probe that runs on every prompt before it is sent. Batch-level gates and the pack audit then check the pack as a whole, for example that real symptoms cannot be told apart from distractors by their data footprint. The simulation kernel that renders the patients is part of this repository; it lives in haenv_kernel/.

• Packs are plugins. Each pack is a judge group that registers its gold blocks, its question template and its scoring; the core pipeline does not know which packs exist.
• One run, one context. An evaluation run carries its own state (RunContext): two runs in one process do not share tables.
• Fingerprints follow definitions. The scoring code and the generation code each have a fingerprint, stamped on every row and every batch. A refactor that only moves definitions is certified definition by definition, so a fingerprint changes when what is computed changes.

The same patients can be presented in four formats (called geometries in the code): single, gated (tests ordered against a budget), slices (independent questions at several time points) and multi (rounds in which the agent may revise). Each pack poses one decision at T; in pack ① the agent may buy tests before it answers.

Scoring

• The gold standard is derived by code from the world that rendered the record. Packs ②, ③ and ④ are scored by code alone. Pack ① is scored by code, and one LLM judge (gpt-6-luna) matches its free-text diagnoses and tests to the gold, with stored votes.

• Guessing scores 0. A pack's decision is scored as chance-corrected balanced accuracy, cc = (m · BA − 1) / (m − 1), with m the answer classes present in the gold: answering at random or always giving the same class scores 0, every item right scores 1.

• One composite per pack.

  Pack	Composite
  ①	mean of the scored dimensions (test F1, diagnosis listed, numerical reading, chance-corrected review), floored at 0, × (1 − hard-gate rate)
  ②	cc of the disposition × (1 − hard-gate rate); a negative cc is never raised by a gate
  ③	cc of the decision, floored at 0, × (1 − hard-gate rate)
  ④	mean of cc of the change source and the skill on the size of the true change, 1 − MAE / MAE_naive (no gate multiplier)

  The packs are separate boards; their composites are not pooled.

• Three columns. Every registered measure is in one of three roles. Scored measures enter the composite. Profile measures judge correctly but separate models too little, or have no validity reading yet; they are computed, stored and shown, and do not enter the composite. Retired measures were found to judge wrongly; they stay in the registry with the reason and are on no board.

• Every score row carries a fingerprint of the judging code, and each pack adds its own. Boards with mixed fingerprints are rejected by the publish gate.

• Raw model responses are stored, so a scoring fix is a recompute (tools/restamp_batch.py <batch>) with no new calls to the models under test.

• verifier_core/ holds the fingerprinting, the publish gate, the hard-gate multiplier, score ceilings and the noise-floor audit. It contains no clinical vocabulary and imports nothing from the clinical layer, so it can be reviewed or reused on its own.

Self-checks run in seconds without model calls; see docs/REPRODUCE.md.

Evaluate your own agent

The model under test is an entry under models: in config.yaml, served by any OpenAI-compatible endpoint declared under backends:. Each turn is one chat request carrying the prompt; an agent behind such an endpoint is evaluated the same way, and its internal tool use is not visible to the judges. It receives the same patients, the same cut at T and the same judges. Plugins are registered explicitly in the job file.

To add an endpoint, put it in config.local.yaml next to config.yaml (merged over it, ignored by git). After pip install, config.yaml is inside the installed package: put the same YAML in a file of your own and name that file in HAENV_CONFIG_OVERLAY. A new backend name registers an OpenAI-compatible backend; its key is read from the file named by HAENV_ENV_FILE:

backends:
  my-endpoint:
    url: http://localhost:8000/v1/chat/completions
    key_env: MY_ENDPOINT_KEY          # MY_ENDPOINT_KEY=... in $HAENV_ENV_FILE
models:
  my-agent:
    backend: my-endpoint
    model: my-agent-v1
    max_tokens: 8000
    price: {input_per_million_usd: 0, output_per_million_usd: 0}   # USD per million tokens

The default models are reached directly: Gemini models through Google's API, Qwen models through DashScope, and every other model through OpenRouter, pinned to one upstream with fallbacks off: the vendor's own endpoint where the account's zero-data-retention setting allows it, otherwise a named host serving the same weights (OpenAI models on Azure, Claude on Google Vertex, MiniMax on Novita). No third-party relay sits between the pipeline and a model.

A billed run counts every request against --judge-budget-usd. On openrouter, google and dashscope the price comes from the provider; on a backend of your own it is the price declared for each model, applied to the token counts the endpoint returns in usage (0 for a free local endpoint). A model there without a price is refused before any request is sent.

uv run haenv run inputs/example-ew.job.yaml --models my-agent --limit 1 \
  --judge-budget-usd 5 --judge-budget-ledger ~/.haenv/budget.json

With --limit the run checks the endpoint on a sample: the semantic dimensions are not judged, no OpenRouter key is needed, and the run exits 6 to say the scores are incomplete. Without it, the semantic dimensions are judged by openai/gpt-6-luna through OpenRouter, so the same env file also needs an OpenRouter key, and the judge's cost counts against the same budget.

examples/ has five plugin packages. Each runs offline and is paired with a negative control. Guides: judge plugins · external task types.

Agent skills

The repository carries three skills under skills/, in the Agent Skills layout (skills/<name>/SKILL.md; the standard is read by more than one agent tool). They are the short path to driving the pipeline from a conversation, and they are written to be read on their own:

Each is an index, not a second source of truth: what it says about the pipeline is checked against the code and the documents linked from it. A skill is a directory of Markdown plus, in the standard's terms, optional scripts/, references/ and assets/; nothing about them is specific to one vendor's tool. If you use an agent that reads the standard, point it at the repository and ask in plain words — "generate a batch of questions from this patient", "run a batch and show me what it cost", "add a judge that catches X" — and the matching skill supplies the conventions, the commands and the failure modes.

Cost and caching

• Generation is cached by model and prompt in cases/_llm_cache/; rebuilding a pack from the same job makes no model calls.
• A run resumes by default and sends only the cells that have no answer yet. A cell the model answered with an empty body, broken JSON or a spent tool budget counts as answered; only aborts caused by the network, a timeout, a provider 5xx or 429, or the key pool are resent. Raw responses are stored, so a judging change is a recompute, not a re-run of the models under test.
• For models with a measured basis, max_tokens must clear a floor derived from their output lengths. This reduces truncation risk but does not guarantee that every answer will finish within budget. batch.json records generation and evaluation usage (gen_usage, eval_usage): measured totals where available, and an explicit status otherwise.
• Scale: a pack item is one request. At N = 50 one pass costs, estimated from each prompt's token count at fitted provider prices, $0.09–$9.55 per model on pack ② (mean $3.20 over the ten 1.x models), $0.09–$8.19 on pack ③ (mean $2.65) and $0.09–$8.62 on pack ④ (mean $2.82); packs ②–④ have no judge cost. Details and the recompute command are in docs/REPRODUCE.md.

Leaderboard

The 1.2.0 board ranks 10 models on the four packs, 50 items each, every item answered 3 times. The overall score is the equal-weight mean of the four pack scores, each from 0 to 1. The 95% intervals come from a case × round bootstrap (4,000 draws that resample items and rounds together); ranks follow the unrounded scores. The board packs are drawn with a private seed; jobs and batches record only its sha256, 8f90c70f61fe7a53860e9a21576c0393e26515ba7eede0467a534360dfa7f9b9.

1.2.0 overall

1.2.0 by pack

Each cell is the pack score with its 95% interval.

Interactive board · Board data (JSON)

1.x board (historical)

The 1.x board ranks ten models on the two diagnosis tracks, ddx-timeline and ddx-workup (145 cases each). It is kept as published and is not comparable with the pack boards: the world, the questions and the scoring have changed since. The bridge items are the only link between the two.

Limitations

• Pack ② does not separate the leading models. Among the six best models on pack ② no pair differs by more than 1.96 standard errors (0 of 15 pairs); pack ③ separates 3 of 15, pack ① 8 and pack ④ 10. On the overall board 9 of the 15 top-six pairs are separable.
• Clinical review. The rules behind packs ② and ③ (which findings send a patient to the emergency department, when a dose change is on target) and several registry entries carry review: pending; the medical content as a whole has not been reviewed by a practising clinician.
• Single judge in pack ①. One model (gpt-6-luna, reasoning high) matches free-text diagnoses and tests to the gold, with two votes and a third on disagreement. The same vendor's models are among those tested; self-preference is not ruled out, and no second judge or physician labelling checks its verdicts. Packs ②–④ have no LLM judge.
• The diagnosis dimensions await validity. On the bridge tracks the diagnosis-track scored dimensions are judgement-based and wait for a clinical blind annotation; until then they have no public column.
• Synthetic data. All patients are synthetic and the benchmark is for evaluation only; nothing here is medical advice.

The full list, with the scoring, generation and world-layer gaps, is in the data card.

Related work

HAEnv combines a longitudinal record, a prompt cut at T, a gold standard fixed before the data, code scoring for every dimension except the semantic ones (one LLM judge) and regenerable synthetic cases. The pass^k reliability metric is from τ-bench (arXiv:2406.12045).

Data, ethics and reproduction

All patients are synthetic. Nothing here is medical advice or suitable for clinical decisions.

• docs/DATA_CARD.md: packs, gold provenance, scoring, known gaps, and why the gold standard ships with the questions.
• docs/REPRODUCE.md: what is free to reproduce, what is billed, and the self-checks.
• docs/ETHICS.md: data sources and limits of use.
• CONTRIBUTING.md: how to change scoring or generation code.

Citation

@software{haenv,
  title  = {Health Agent Environment},
  author = {{Theta Health}},
  year   = {2026},
  url    = {https://github.com/thetahealth/mirobody-env},
  version = {1.2.1}
}

The same metadata is in CITATION.cff.

License and acknowledgements

Code is under the MIT License; the synthetic data and question packs are under CC BY 4.0. Third-party notices are in NOTICE.md.

HAEnv is inspired by ESL-Bench.

_{Synthetic data · evaluation use only · not medical advice}