medagentrx 0.1.0

The prescription for self-improving medical AI.
Every model update, by an engineer, a pipeline or an AI agent, is checked against its approved change plan (FDA PCCP, EU AI Act Art. 43(4)).

Clinical risk models are increasingly retrained by scheduled pipelines and by AI agents. Regulators allow this only for changes agreed in advance, in a Predetermined Change Control Plan in the US or under Article 43(4) of the EU AI Act. MedAgentRx turns that plan into a test that runs on every update.

The plan is a single YAML file. Each candidate update is evaluated against it and receives one verdict: ACCEPTED, FAILED or OUTSIDE_PLAN. Every evaluation, including every failure, is appended to a hash-chained ledger, and the plan's limits are read back from it. Each attempt uses up part of the plan's budget of looks at the locked test set. A model therefore cannot be selected simply because it passed once among many tries.

Each verdict rests on established methods. AUROC is compared with paired DeLong tests, cross-checked against R pROC. Calibration is judged by slope and intercept, and clinical value by net benefit at the plan's decision thresholds. Every subgroup must be non-inferior on its own. Alpha is spent across the life of the plan, and each update is also compared with the originally authorised version, which catches slow drift. Methods gives the formulas and references.

Five updates, each non-inferior to the one before. The fifth exceeds the plan's drift budget against the authorised version and fails.

What it checks

The checks catalogue lists all 19 checks with their regulatory cross-references.

Quick start

pip install medagentrx
medagentrx demo

The demo runs ten scenarios on a synthetic cohort, with no real patient data. Each scenario plants one defect, or none, and must produce exactly its expected verdict. To check your own update:

medagentrx init                                   # write a commented change-plan.yaml
medagentrx sample-size change-plan.yaml --auroc 0.80 --prevalence 0.12
medagentrx fingerprint test-predictions.parquet   # fingerprint the locked test set
medagentrx check change-plan.yaml \
  --candidate candidate.parquet --previous previous.parquet --anchor anchor.parquet \
  --manifest train_manifest.parquet --model-meta model.yaml \
  --ledger ledger.jsonl --out report/

The exit code is the verdict: 0 ACCEPTED, 1 FAILED, 2 OUTSIDE_PLAN, 3 input error. report/ holds a self-contained HTML report, JSON, a Markdown summary and JUnit XML. MedAgentRx reads prediction files only and never loads a model binary.

The plan

modifications:                 # FDA Description of Modifications
  - {id: M1, type: retrain_same_architecture, automatic: false, scope: global}
data_management:
  training_inclusion:
    all:
      - {field: age, op: ge, value: 18}
      - {field: site, op: in, value: [A, B, C]}
  locked_test_set: {sha256: "e9a46f9b...", max_evaluations: 20}
  forbid_overlap_on: [patient_id]
performance:
  alpha_total: 0.05
  alpha_spending: bonferroni
  criteria:
    - {metric: auroc, test: non_inferiority, vs: [previous, anchor], margin: 0.02}
    - {metric: calibration_slope, test: within, bounds: [0.8, 1.2]}
  cumulative_drift: {metric: auroc, max_drop_vs_anchor: 0.02}
  subgroups: {by: [sex, age_band, site], metric: auroc, max_drop: 0.03, min_events: 30,
              on_insufficient: fail, multiplicity: none}

The full example is examples/sepsis-risk/change-plan.yaml. medagentrx render-plan turns the same file into the PCCP document, with the FDA components and the EU AI Act cross-reference, so the document and the gate cannot drift apart.

Automated retraining and agents

from medagentrx.guard import Guard

guard = Guard("change-plan.yaml", "ledger.jsonl", mode="thresholdout", previous="previous.parquet")
decision = guard.submit("candidate.parquet", "model.yaml", manifest="train_manifest.parquet")
print(decision.allowed, decision.reason, decision.looks_remaining)

An agent may try only the modifications the plan marks as automatic. Each try spends one look at the locked test set. In the default thresholdout mode the agent learns only whether the candidate passes, with random noise added so that repeated tries reveal little about the test set. If the agent also supplies predictions on its own validation data, the guard follows Thresholdout (Dwork et al. 2015) and answers from those whenever they agree with the test set. See For agents and AutoML.

Continuous integration

GitHub Actions:

- uses: a7med7emedan/medagentrx@v0.1.0
  with:
    plan: change-plan.yaml
    candidate: predictions/candidate.parquet
    previous: predictions/previous.parquet
    anchor: predictions/anchor.parquet
    manifest: data/train_manifest.parquet
    model-meta: model.yaml
    ledger: governance/ledger.jsonl

GitLab CI/CD:

include:
  - component: $CI_SERVER_FQDN/<group>/medagentrx/medagentrx@0.1.0
    inputs:
      image: <registry>/medagentrx@sha256:<digest>
      plan: change-plan.yaml
      # same inputs as above

There is also a medagentrx-validate pre-commit hook. See the CI guides.

Verdicts

Warnings never change the verdict. They always appear in the report.

Validation

The simulation grid spans prevalence 5 to 30 percent and test sets of 500 to 20,000 rows, with 20,000 replicates per cell for type I error and 10,000 for coverage. At a nominal one-sided 5 percent, the non-inferiority test rejects a true boundary null 4.7 to 5.4 percent of the time. Nominal 95 percent bootstrap intervals cover 93.3 to 95.0 percent. DeLong results match R pROC to 1e-8 on AUC and 1e-6 on standard error and z. See validation results and methods.

Security

Inputs are data only: no unpickling, no eval, no user code. Plans are parsed with a hardened safe YAML loader and strict schemas. Reports are self-contained and load nothing. The container runs as a non-root user with a read-only file system, and has no HIGH or CRITICAL Trivy findings. Report vulnerabilities privately as described in SECURITY.md.

Citing

If you use MedAgentRx in research, please cite it using CITATION.cff.

Disclaimer

MedAgentRx supports a quality management system. It is not legal or regulatory advice, does not replace your QMS or regulatory strategy, and its output is not itself a regulatory submission.

Author

Ahmed Hemedan. Licensed under Apache-2.0.