Healthcare Escalation Bench
A benchmark for one question: when the workflow state changes, does an LLM agent still know whether to answer, ask, abstain, or hand the case to a human?
Each of the 400 synthetic episodes is a short healthcare workflow: an earlier record, newer information, and a request. The same scenario appears in five controlled variants, so you can see exactly which kind of change breaks an agent.
| Variant | What changes |
|---|---|
clean |
All needed information is present |
missing |
A decision-critical fact is removed |
stale |
An outdated chart summary appears after newer facts |
conflict |
A second source contradicts the record, or the patient minimizes a concerning finding |
late_risk |
A risk cue arrives after the request was submitted |
The agent picks one of four actions: ANSWER, ASK_CLARIFICATION, ABSTAIN,
or ESCALATE. Episodes cover four tasks: documentation updates, evidence
retrieval, patient messaging, and triage handoff.
This is the companion artifact of Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures (arXiv:2610.21270).
Scope. No real patient data is included. Reference actions follow one documented study policy written by one author (docs/reference_policy.md). They are not clinician-adjudicated. Results measure agreement with that policy under controlled changes. They do not show clinical safety or deployment readiness. See Limitations.
Install
Python 3.10 or newer. The core package has no dependencies.
python -m pip install healthcare-escalation-bench
escalation-bench info
To work from source or reproduce the paper, clone the repository:
git clone https://github.com/Garyouki/healthcare-escalation-bench.git
cd healthcare-escalation-bench
python -m pip install .
escalation-bench info
Evaluate your own agent
Your data and your model stay on your machine. Nothing is uploaded.
Option A: you already have an agent pipeline.
-
Export the inputs. They contain only
episode_id,task, andtranscript:escalation-bench export --split test --out inputs.jsonl
-
Send each
transcriptto your agent. Do not includeepisode_idin the prompt, because the id names the variant.escalation-bench promptprints the system prompt used in the paper, if you want a common starting point. -
Save one JSON line per episode:
{"episode_id": "doc_03_p0_conflict", "action": "ESCALATE", "confidence": 0.82}
confidenceis optional. A CSV with the same columns also works. -
Score it and write a report:
escalation-bench score predictions.jsonl --name "My agent" --report report.md
Option B: run a model directly. The runner uses only the standard library.
# Local Ollama server (the paper's setup)
escalation-bench run --model qwen3:4b --out preds.jsonl
# Any OpenAI-compatible endpoint: vLLM, LM Studio, llama.cpp, or a hosted API
export OPENAI_API_KEY=...
escalation-bench run --backend openai --base-url http://localhost:8000/v1 \
--model my-model --out preds.jsonl
escalation-bench score preds.jsonl --name "My model" --report report.md
Runs are resumable. Use --limit 20 for a quick check first.
What the report contains
- Decision accuracy, escalation sensitivity and specificity, missed-escalation rate, and cost-weighted utility, with 95% confidence intervals that resample whole scenarios.
- Results by variant and by task, so you can see which state change hurts.
- A reference-action → chosen-action table.
- The missed escalations, most confident first, with the risk cue the agent missed.
- Calibration of stated confidence, when you supply it.
- The paper's five models on the same split, for context.
Missing, invalid, or duplicate predictions are listed, never silently dropped. examples/sample_report.md shows a full report.
Metrics
- Decision accuracy: the chosen action is in the episode's acceptable set. 62 of the 400 episodes accept more than one action.
- Escalation sensitivity / specificity / missed-escalation rate: binary
ESCALATEagainst the single reference action. - Utility at k:1: per episode, +1 for each correct escalate or non-escalate decision, −1 for an unnecessary escalation, and −k for a missed one. The paper reports k = 5 and k = 10. These costs are modeling choices, not measured clinical harms.
- Calibration: 10-bin ECE, Brier score, and AUROC of (1 − confidence) for predicting a wrong decision.
Dataset
| Episodes | Seeds | Escalation required | |
|---|---|---|---|
| dev | 80 | 8 | 30 |
| test | 320 | 32 | 100 |
| all | 400 | 40 | 130 |
40 hand-written scenario seeds × 2 surface wordings × 5 variants. Episodes are
assembled by code, not generated by a language model. escalation-bench verify-data rebuilds every episode from the seeds and checks the SHA-256
against the manifest.
Each episode record in src/escalation_bench/data/episodes.jsonl includes the
reference action, the acceptable set, the risk cues, the required facts, and a
rationale, so every label can be inspected or replaced.
Reproduce the paper
The raw outputs of the paper's five models (Ollama, temperature 0) are in
results/run_combined_5models/raw/. To regenerate every table and figure:
python -m pip install ".[reproduce]"
python reproduce/evaluate.py results/run_combined_5models
python reproduce/audit_analysis.py results/run_combined_5models
python reproduce/review_revision_analysis.py .
python reproduce/figures.py results/run_combined_5models
All table values match the published results. tests/test_reproduce.py
checks the main table against the paper. One note: risk-coverage curves sort
by confidence with NumPy's default (unstable) sort, so the order of tied
confidences, and the shape of those curves between ties, can differ across
NumPy versions. Curve endpoints and all tabulated metrics are unaffected.
results/source_manifests/ records model digests, decoding settings, and the
dataset hash for the two source runs.
Limitations
- Labels are one author's study policy. No clinician annotation, agreement statistic, or adjudication exists yet.
- Small and synthetic. 32 independent test scenarios. Wording was written to probe mechanisms, not sampled from real clinical communication.
- Model scope. The paper evaluates five small open-weight models. No frontier or clinically adapted model has been evaluated yet.
- Detectors are lexical. The paper's controller uses transparent regex detectors. They catch familiar trigger phrases but miss unseen paraphrases.
Help wanted
These are the open items from peer review. Contributions are very welcome.
- Clinician review. A blinded review of a subset of episodes by two independent clinicians, with agreement analysis and adjudication.
- Independent episodes. New scenarios written by someone other than the original author, to test whether the findings hold.
- Stronger models. Reports from frontier or clinically adapted models.
- Better detectors. Detectors that work on paraphrased language, tested end to end.
If you run the benchmark on your own system, please open an issue with your report or email the maintainer. See CONTRIBUTING.md.
Citation
@misc{zheng2026auditingescalationreliabilityhealthcare,
title = {Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures},
author = {Zihao Zheng and Junyi Yao and Baichuan Li and Jiayu Long},
year = {2026},
eprint = {2610.21270},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.21270}
}
License
Code: MIT. Dataset (episodes, seeds, and bundled model outputs): CC BY 4.0.
Metadata
Release files for healthcare-escalation-bench 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| healthcare_escalation_bench-0.1.0.tar.gz | 116.2 kB | With provenance | Details for healthcare_escalation_bench-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| healthcare_escalation_bench-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for healthcare_escalation_bench-0.1.0-py3-none-any.whl |
Total release size: 235.6 kB
Release files / healthcare_escalation_bench-0.1.0.tar.gz
| Download URL | healthcare_escalation_bench-0.1.0.tar.gz |
|---|---|
| Size | 116.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
28086d38fa091486f81ffb7c300f6516662134103bd2d7c513576b0485152a17
|
|
BLAKE2b-256 checksum How to use checksums |
82158c35f0da794748de6b3e4fcd9bcd388c446079e432b96f41464ef5222fe7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / healthcare_escalation_bench-0.1.0-py3-none-any.whl
| Download URL | healthcare_escalation_bench-0.1.0-py3-none-any.whl |
|---|---|
| Size | 119.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e4662ed092d39797e93224d920d0465be33aedf26c88d33f543351852330c406
|
|
BLAKE2b-256 checksum How to use checksums |
bc7bc71c14ba216318363300790e43795bfa81bf819db91959d73871351e06c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logSecurity for healthcare-escalation-bench 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for healthcare_escalation_bench-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.