Health-DeID

Health-DeID is local research software for auditable, resumable de-identification of clinical text and selected structured fields. Its desktop web UI, CLI, and Python API all use the same SQLite run database.

⚠️ Important: Health-DeID has not been validated or approved for diagnosis, treatment, clinical decisions, or as a guarantee of legal or regulatory compliance. Validate it for your data and use case before sharing output.

Features

• Imports JSONL or Parquet while preserving input order and selected metadata.
• Detects PHI with AWS Comprehend Medical, Claude Sonnet 4.6 on Amazon Bedrock, local rules, or a combination.
• Retains, redacts, generalizes, date-shifts, or replaces each PHI category.
• Generates replacements with Faker or a custom list, consistently by entity, record, or occurrence.
• Applies the same replacement policy to unstructured text and mapped structured fields.
• Optionally validates de-identified text with GPT-OSS Safeguard 120B and includes a human review user interface.
• Recovers interrupted work and retains completed backend requests.
• Creates revised runs that reuse detector results when the input and detector setup are identical, while rerunning rules and downstream de-identification.
• Exports explicitly selected columns to JSONL or Parquet.

Requirements

• Python 3.12 through 3.14
• AWS credentials and access to the configured services for AWS-backed runs

Installation

pip install health-deid
health-deid --version

AWS services use boto3's standard credential and region resolution; credentials are not entered or stored in the UI.

Start with the UI

health-deid ui --reviewer "Reviewer Name" --runs-dir runs

Open one run directly:

health-deid ui runs/RUN_ID --reviewer "Reviewer Name"

CLI

The CLI intentionally has eight commands:

health-deid check config.yaml
health-deid run config.yaml
health-deid status runs/RUN_ID
health-deid resume runs/RUN_ID
health-deid retry runs/RUN_ID --failed
health-deid revise runs/RUN_ID revised.yaml --reason "change date handling"
health-deid export runs/RUN_ID final.parquet \
  --mode ready_only --column record_id --column final_text
health-deid ui --reviewer "Reviewer Name"

check is a precheck: it validates setup and shows an estimated AWS cost without making paid model calls. After processing starts, status and UI usage cards show actual costs.

Run data

Normal operation creates:

runs/<run-id>/
├── run.sqlite
└── exports/

The database contains source clinical text, identifiers, backend responses, findings, transformations, review decisions, usage, errors, and export history. Treat the complete run directory as PHI even after creating a de-identified export.

The processing sequence is:

input → detection and rules → draft de-identification → validation → review → final de-identification

Configuration

A minimal Comprehend configuration is:

config_version: 1
run:
  name: example
  output_dir: runs
input:
  path: notes.jsonl
  format: jsonl
  record_id_column: note_id
  entity_id:
    source: column
    column: patient_id
  text_column: note_text
detection:
  enabled: true
  detectors:
    - backend: aws_comprehend_medical
      region_name: us-east-1
      cost_per_100_characters_usd: 0.0014
  execution:
    workers: 2

The supported Bedrock detector and validator models are fixed:

• us.anthropic.claude-sonnet-4-6
• openai.gpt-oss-safeguard-120b

AWS calls use botocore's standard retry mode. Detection and validation each expose only the useful local concurrency setting, workers, which defaults to 2. Validator responses that are explicitly truncated are retried with the next fixed output-token budget.

Non-date synthetic replacements use one of two direct methods:

• faker: realistic values generated by Faker.
• custom_list: values supplied in configuration or the UI.

Both support entity, record, and occurrence consistency. Date replacement uses date_shift, with one whole-week offset per entity. Dates can also be generalized to the year, and ages can be generalized to 90+.

Python API

from health_deid import create_run, open_run

run = create_run("config.yaml").execute()
print(run.status())

reopened = open_run(run.run_dir).resume()
revised = reopened.revise(
    "revised.yaml",
    reason="change replacement policy",
)
revised.export(
    output_path=revised.context.exports_dir / "final.jsonl",
    format="jsonl",
    mode="ready_only",
    selected_columns=["record_id", "final_text"],
)

A revised run reuses paid detection only when the imported input and detector definitions match. Use rerun_detection=True (or CLI --rerun-detection) to authorize a new detector pass when they do not.

Examples

See the examples for thirteen configurations using fictional records.

Documentation

See the documentation for the full user guide, API, and module reference (generated with Sphinx).

Acknowledgements

• Developed by: Omid Jafari
• Research affiliation and acknowledgement: Ang Li Lab

License

Health-DeID is licensed under the GNU Affero General Public License v3.0 or later.