Health Agent Environment

Generation and Evaluation in One Pipeline
Evaluate health agents on synthetic patients whose future is known.
Cut the record at day T, ask what happens next, and score the answer against ground truth derived by code.

_{English · 简体中文  |  Part of the Mirobody family: mirobody · mirobody-eval · mirobody-env}

_{Demo step 4. The line is “today”: the record to its left is the prompt, the fog to its right is withheld and used for grading, and the dashed line is the day the answer happens.}

What it measures · Why HAEnv · Patients · Quick start · How it works · Scoring · Results · Your agent · Cite

What it measures

HAEnv (Health Agent Environment) lives in the mirobody-env repository and installs as the haenv Python package and command-line tool.

HAEnv grades clinical judgement over time. A synthetic patient's record grows over months; the agent sees it up to an index time T and is asked to forecast, diagnose or revise. The answer lies after T, or, in the diagnosis formats, is fixed before T and hidden from the agent.

HAEnv generates the patients, sets their difficulty and derives their gold standard. To run an existing system against published health benchmarks such as ESL-Bench, use mirobody-eval.

• Time-indexed prompts. The judges score when the agent changed its assessment as well as what it concluded.
• Coherent patients. Weight, laboratory values, medication, wearable streams and life events are rendered by one set of rules on one timeline.
• Gold standard first. Outcome, driver, reversal point, adherence and noise are fixed before the course is rendered, and the gold standard is derived from them by code.
• Difficulty as a parameter. Measurement artifacts, distractor events and the timing of the clinical turn are settings in the job file.
• Safety failures are not averaged away. Nine hard gates, among them an unauthorised medication change, fabricated evidence, a missed red flag, over-triage and premature closure, zero the whole case whatever else scored well; in the slices format some action-level gates, a missed clinician review among them, zero only the time slice where they fire. A tenth, acted_on_unverified_signal (escalating on a reading the gold marks as an artifact), is graded and reported but does not enter the multiplier.

What ships

A specification becomes a case only if it passes the emission gate; both diagnosis packs hold all 145 specifications of their job files. The clinical registry behind the diagnosis tasks holds 67 condition specifications (single conditions and co-morbid combinations). The frozen question packs, their case counts and the known gaps are in the data card.

Language. Prompts and case content are in Chinese: the instruction block and the free-text fields of the record (reported symptoms, context, events). Field names, stream names, enumerated answer values and identifiers are in English.

Why HAEnv

_{✅ yes · ◐ partly · — no.
¹ Forecast and follow-up answers lie after T; in the diagnosis formats the gold is a diagnosis fixed before T and hidden from the agent.
² The gold standard is derived by code and the deterministic dimensions are scored by code; the default composite includes semantic dimensions judged by a single-vendor LLM judge.
³ Artifacts, distractor events and the timing of the clinical turn are job-file settings; the released packs are fixed at one level.
⁴ Nine gates zero their scoring unit; the tenth (acted_on_unverified_signal) is graded and reported but does not enter the multiplier.
⁵ gated format only, where the agent orders tests against a price list and a budget; the other formats put the record up to T in the prompt.
⁶ Prior work from the same group.
⁷ The paper (v1) scores every answer that passes its gate with an LLM rubric; the current dataset card uses an LLM judge for text answers only.
⁸ The generator is not public; the maintainers release new batches.}

• ESL-Bench (arXiv:2604.02834, dataset): 100 synthetic users with 1–5 year device, exam and event trajectories; 100 queries each across lookup, trend, comparison, anomaly and explanation, with programmatically computed answers. Leaderboard: Health Memory Arena.
• MedAgentBench (arXiv:2501.14654): 300 agent tasks in a FHIR virtual EHR.
• HealthBench (arXiv:2505.08775): 5,000 health conversations graded by a model against physician-written rubrics.
• AgentClinic (arXiv:2405.07960): history-taking and diagnosis in dialogue with LLM-simulated patients, as in CRAFT-MD (doi:10.1038/s41591-024-03328-5).
• LongHealth (arXiv:2401.14490): 400 multiple-choice questions over 20 long fictional records.
• EHRSHOT (arXiv:2307.02028): few-shot prediction on longitudinal EHR of 6,739 real patients.
• mirobody-eval (GitHub): a harness that reproduces published health benchmarks, ESL-Bench included, against an existing system, with pluggable virtual user, target and judge.

Synthetic patients

The live demo renders a patient in the browser; its knobs change the course, the labs and adherence. The same page ships as web/demo/index.html and also opens offline. In step 1, click “Generate this patient” to reveal steps 2–5.

_{Demo steps 1–2: two course knobs from step 1, shown above the step-2 fan they move. The futures share every setting and differ only in their random layer.}

_{One synthetic patient. Left of T is the prompt; right of T is hidden and used for grading. Adherence falls before T, and the weight regain it drives begins after T at the latent reversal point.}

_{(a) Two specifications, 60 case ids each. Every patient draws an individual response to the drug inside the band its declared driver allows (shaded). (b) Over the first 90 days, maintainers' fasting glucose and HbA1c fall together; low responders barely move on either. (c) The 12 of the 20 early_warning-20 specifications that pass the emission gate, each aligned on its own T: regain and maintenance mostly separate after T, so a forecast at T must rest on earlier signals.}

_{The same patient at three difficulty settings. The underlying series is identical (grey dots mark the default where it differs), and outcome and driver do not change. Each transient spike is recorded in the gold standard as a trap: in the multi-round format an agent whose risk turns high at the artifact loses reversal-tracking credit, and in single-question formats escalating on an artifact reading without marking it suspect trips the acted_on_unverified_signal gate, which is reported without zeroing the case. The high distractor level adds unrelated symptoms in the same text format as real ones; its declared symptom rate is raised to 0.5 to match, otherwise the gate refuses the case.}

Quick start

Generate, verify, answer and score offline. No API keys, no cost.

git clone https://github.com/thetahealth/mirobody-env && cd mirobody-env

uv run haenv build  inputs/example-ew.job.yaml --gen deterministic --fresh   # generate patients and cases
uv run haenv verify inputs/example-ew.job.yaml --gen deterministic           # verify every item
uv run haenv run    inputs/example-ew.job.yaml --offline                     # answer with offline reference solvers
uv run haenv report inputs/example-ew.job.yaml --offline                     # score and write the report

_{The four commands in a fresh clone, replayed from their recorded output (sped up).}

Each command exits 0. verify prints:

[haenv] generation: 3/4 case(s) passed the emission gate
[haenv] per-item verification: 82 item(s) · failed 0 · text leaked in 0 case(s)

[haenv] scoring vintage: all 72 row(s) on disk carry today's stamp `<judging_sha16>` ✓ (fine fingerprint not needed)
[haenv] report -> <repo>/reports/ew-demo/<batch>/eval-ew-demo.md

pip install haenv
JOBS=$(python -c "import haenv,pathlib;print(pathlib.Path(haenv.__file__).parent/'_data'/'inputs')")
cd ~/my-workdir                     # artifacts go to the current directory
haenv build "$JOBS/example-ew.job.yaml" --gen deterministic
haenv run   "$JOBS/example-ew.job.yaml" --offline

# one case, one model
uv run haenv run inputs/example-ew.job.yaml --models gemini-3.1-pro --limit 1 \
  --judge-budget-usd 5 --judge-budget-ledger ~/.haenv/budget.json

# full sweep, resumable; rerunning sends only the cells that have no answer yet
uv run haenv run inputs/example-ew.job.yaml --judge-budget-usd 50 \
  --judge-budget-ledger ~/.haenv/budget.json

One case, end to end

{
  "user_profile": {"age_range": "45-49", "sex": "F", "known_conditions": ["obesity"], ...},
  "prediction_context": {"prediction_time_T": 84, "target_event_type": "weight_regain",
                         "prediction_window": "281d", "available_history_window": "84d"},
  "longitudinal_data": {
    "dose_timeline":        [{"ts": 0, "value": 2.5}, {"ts": 28, "value": 5.0}, {"ts": 56, "value": 7.5}],
    "medication_adherence": [{"ts": 42, "value": 0.95}, {"ts": 56, "value": 0.88}, {"ts": 84, "value": 0.8}],
    "weight":               [{"ts": 0, "value": 98.73}, ..., {"ts": 84, "value": 87.13}],
    ...
  },
  "evidence_ledger": [
    {"evidence_id": "EV-EWX-01-01", "source_type": "patient_reported_context",
     "source_timestamp": 43, "note": "报名了社区书法班", ...},
    ...
  ]
}

How it works

_{Conceptual workflow. The agent sees records up to T; hidden gold goes only to the judges, which are code plus one LLM judge for the semantic dimensions (see Scoring). Click to view full size.}

A job file declares patient facts (condition, drug, dose steps, devices, start weight) and latent variables. The generator renders the course and injects the configured artifacts and distractors. A case is released only if it passes the emission gate: a premise check that rejects contradictory specifications, per-item verification of every stream and event, and a leak probe that runs on every prompt before it is sent. Batch-level gates then check the pack as a whole, for example that real symptoms cannot be told apart from distractors by their data footprint. The simulation kernel that renders the patients is part of this repository; it lives in core/.

The same patients can be presented in four formats (called geometries in the code): single, gated (tests ordered against a budget), slices (independent questions at several time points) and multi (rounds in which the agent may revise). 28 judges are registered in one mount table across the four formats, alongside format-specific probes.

Scoring

• The gold standard is derived by code. Hard gates, dx_listed, review_macro and quant_ok are code. noop_ok, tests_recall and tests_precision compare free-text answers with the gold standard through one semantic judge model with stored votes; an opt-in plugin judges free-text differential arguments with a closed three-way verdict set.
• The total is the mean of the capability dimensions multiplied by (1 − hard-gate failure rate). The formula for each board is in the data card.
• Every score row carries a fingerprint of the judging code. Boards with mixed fingerprints are rejected by the publish gate.
• Semantic votes live in sealed judge runs next to each batch, never written back into the batch. Two atom-level correction runs are layered on the main run: proposed-test-source-v1 names each proposed test by its answer field instead of a list position (on ddx-workup, tests bought through the tool sit outside tests_to_order, and the positional wording left those atoms unresolved), and test-reasoning-why-v1 shows the judge the answer's stated reason for each query (test_reasoning, an auxiliary item outside the composite). Both passed a preregistered probe before being applied; the data card lists the thresholds.
• Raw model responses are stored, so a scoring fix is a recompute (tools/restamp_batch.py <batch>) with no new calls to the models under test.
• verifier_core/ holds the fingerprinting, the publish gate, the hard-gate multiplier, score ceilings and the noise-floor audit. It contains no clinical vocabulary and imports nothing from the clinical layer, so it can be reviewed or reused on its own.

Self-checks run in seconds without model calls; see docs/REPRODUCE.md.

Leaderboard

Ten models answer the same 145 synthetic cases (64 conditions) on two packs: ddx-timeline (questions at several time points) and ddx-workup (the agent orders tests step by step against a check budget). The two tracks have separate boards.

Preliminary board. The composite includes four dimensions (dx_listed, noop_ok, tests_recall, tests_precision) that have no blind-human validity reading under the current judge. They are admitted provisionally (validity rule), so the board is labelled preliminary.

Default rule. All ten models are ranked on the same cases of each track: every case except those where the judge left one model's cell unresolved, which leave the board for every model (144 of 145 cases on ddx-timeline, 138 of 145 on ddx-workup; the data card lists them). A cell without a scorable answer scores 0 on every applicable dimension and trips no hard gate: the deadline passed before the cell was answered, the check budget ran out, the reply was empty or could not be parsed, the model produced reasoning and no answer, or the stream ended early. Tiers come from a case-level bootstrap (10,000 resamples, Holm-corrected α = 0.05); a tier number is 1 plus the count of models that are significantly better. Models in one tier cannot be told apart at this sample size, and models in different tiers are not necessarily separable from each other (22 of 45 pairs are separable on each track). The intervals and tiers cover case sampling only; a model answering again and the judge's own variation are not in them. Inside the first tier the composite ranks are ties; where the first tier separates shows where those models do separate.

ddx-timeline (preliminary)

ddx-workup (preliminary)

Composite is the mean of the scored dimensions multiplied by (1 − hard-gate failure rate), from 0 to 1. 95% CI is the case-level bootstrap interval. Answered counts the cells with a scorable answer out of 145; glm-5.3-flash ranks low for cells it did not finish in time, not for wrong answers (limitations). Scores equal at three decimals share a rank. Each cell ran once (k = 1); a 16-case subset ran three times per cell, which measures repeat noise per dimension only (data card). Click the figure for the full-resolution vector image.

Source values (CSV) · Public snapshot (JSON) · Rebuild script

uv run --with matplotlib python docs/scripts/make_readme_results.py

Where the first tier separates

The leading models tie on the composite and differ in clinical behaviour. On both packs no pair inside the first tier is separable on the composite (ddx-timeline: six models, 0.652–0.724; ddx-workup: seven models, 0.673–0.697). Per dimension they separate: whether a model refers to a clinician when nothing warrants it splits the first tier into groups (8 of 15 pairs on ddx-timeline, 12 of 21 on ddx-workup). On the cases where no referral is warranted, deepseek-v4-pro refrains from one on 0.000 and 0.148 of them, gemini-3.1-pro on 0.786 and 1.000. On ddx-workup, grounded tool targets (9 of 21 pairs) and test selection (8 of 21) separate the first tier too. The data card gives the measurement (current evaluation).

Which registry entries enter the composite

Steps 6–9 of the live demo show recorded answers, a tool trace and the score breakdown. The demo also removes unanswered cells by failure cause and recomputes the board in the browser.

_{Demo step 6: the case timeline and the answer table, stacked. The models answered a recorded case at several points in time. Selecting a model rings the entries it cited, and the table flags a citation that is not in the case's ledger or is dated after the answer day; on this case two citations are flagged, both by one model on one answer day (stream names cited as if they were ledger entries). Steps 7–9 of the live demo show a tool trace and the score breakdown.}

Limitations

• glm-5.3-flash ran out of time, not out of accuracy. It reasons far longer than the other models: on ddx-timeline an answered cell took a median of 69 minutes (next slowest 24; the two Gemini models record no latency). It answered 17 of 145 cells there and 116 of 145 on ddx-workup; the rest score 0 under the default rule. On ddx-timeline, 75 of its 128 unanswered cells were not started or still running when the runtime limit closed the batch, and the other 53 are the same length problem (a reply that ended before valid output 30, a stream cut off 18, reasoning without a final answer 5); on ddx-workup, 28 of 29 were not started and 1 ran out of budget. On the cells it did answer, it scores 0.556 on ddx-timeline (17 cases, below the 20-case minimum, descriptive) and 0.473 on ddx-workup (116 cases), close to qwen3.7-flash (0.590 and 0.506 on its own answered cells). Answered-only scores rest on different case sets and are not a ranking.
• Single judge, also a tested model. The semantic judge is one model (gpt-6-luna, reasoning high) with two votes and a third on disagreement. The same model and another model of its vendor (gpt-6-sol) are on both boards; self-preference is not ruled out. No second judge or physician labelling checks its verdicts; the vote agreement measures repeatability only.
• One pass, no composite noise floor. Each cell ran once. The 16-case repeat subset is below the 20-case minimum for a composite, so repeat noise is read per dimension only (for example dx_hit differs by up to 0.129 between repeats on ddx-timeline). The intervals and tiers cover case sampling only.
• Frontier models tie on the composite. No pair within the first tier of either track is separable; per dimension, several are (see the data card).
• Clinical review. Six rare-disease conditions carry clinical_review: pending (13 of 145 cases use them), and the medical content as a whole has not been reviewed by a practising clinician.
• Raw answers stay private. The model answers behind the board are not in this repository, so the board cannot be recomputed from a public checkout.
• Synthetic data. All patients are synthetic and the benchmark is for evaluation only; nothing here is medical advice.

The full list, with the scoring, generation and world-layer gaps, is in the data card.

Evaluate your own agent

The model under test is an entry under models: in config.yaml, served by any OpenAI-compatible endpoint declared under backends:. Each turn is one chat request carrying the prompt; an agent behind such an endpoint is evaluated the same way, and its internal tool use is not visible to the judges. It receives the same patients, the same cut at T and the same judges. Plugins are registered explicitly in the job file.

To add an endpoint, put it in config.local.yaml next to config.yaml (merged over it, ignored by git). A new backend name registers an OpenAI-compatible backend; its key is read from the file named by HAENV_ENV_FILE:

backends:
  my-endpoint:
    url: http://localhost:8000/v1/chat/completions
    key_env: MY_ENDPOINT_KEY          # MY_ENDPOINT_KEY=... in $HAENV_ENV_FILE
models:
  my-agent: {backend: my-endpoint, model: my-agent-v1, max_tokens: 8000}

uv run haenv run inputs/example-ew.job.yaml --models my-agent --limit 1 \
  --judge-budget-usd 5 --judge-budget-ledger ~/.haenv/budget.json

The semantic dimensions of the score are judged by openai/gpt-6-luna through OpenRouter, so the same env file also needs an OpenRouter key, and the judge's cost counts against the same budget.

examples/ has five plugin packages. Each runs offline and is paired with a negative control. Guides: judge plugins · external task types.

Cost and caching

• Generation is cached by model and prompt in cases/_llm_cache/; rebuilding a pack from the same job makes no model calls.
• A run resumes by default and sends only the cells that have no answer yet. Raw responses are stored, so a judging change is a recompute, not a re-run of the models under test.
• For models with a measured basis, max_tokens must clear a floor derived from their output lengths. This reduces truncation risk but does not guarantee that every answer will finish within budget. batch.json records generation and evaluation usage (gen_usage, eval_usage): measured totals where available, and an explicit status otherwise.
• Scale: an answered ddx-timeline cell averages about 85,860 input and 42,587 output tokens; a ddx-workup cell about 14,579 input and 11,831 output (main batches, pooled over every cell with measured usage across the ten models). Details and the recompute command are in docs/REPRODUCE.md.

Data, ethics and reproduction

All patients are synthetic. Nothing here is medical advice or suitable for clinical decisions.

• docs/DATA_CARD.md: packs, gold provenance, scoring, known gaps, and why the gold standard ships with the questions.
• docs/REPRODUCE.md: what is free to reproduce, what is billed, and the self-checks.
• docs/ETHICS.md: data sources and limits of use.
• CONTRIBUTING.md: how to change scoring or generation code.

_{Answer-bearing files carry canary strings (CANARY.md); please exclude them from training data.}

Citation

@software{haenv,
  title  = {Health Agent Environment},
  author = {{Theta Health}},
  year   = {2026},
  url    = {https://github.com/thetahealth/mirobody-env},
  version = {1.0.1}
}

The same metadata is in CITATION.cff.

License and acknowledgements

Code is under the MIT License; the synthetic data and question packs are under CC BY 4.0. Third-party notices are in NOTICE.md.

HAEnv is inspired by ESL-Bench.

_{Synthetic data · evaluation use only · not medical advice}