Medical Data Validator

A Python library and REST API for validating healthcare datasets — PHI/PII detection, HIPAA/GDPR/FDA compliance scoring, data quality checks, and interactive dashboards.

Scope notice: This tool assists research and data engineering teams in assessing data quality and identifying compliance risks. It is not a substitute for a certified HIPAA/GDPR audit, legal counsel, or a Business Associate Agreement. Regulatory compliance requires organisational controls, staff training, and third-party assessment that go beyond software.

Features

Core Validation

• Multi-format support: CSV, Excel, JSON, Parquet
• PHI/PII detection: All 18 HIPAA Safe Harbor identifiers
• Medical code validation: ICD-10, LOINC, CPT
• Data quality checks: Completeness, accuracy, consistency, timeliness
• Extensible rule system: Custom ValidationRule subclasses + setuptools entrypoints

Compliance & Analytics (v1.2)

• Compliance scoring: HIPAA, GDPR, FDA 21 CFR Part 11 (heuristic, not certified)
• FHIR R4 & SNOMED CT plugins: via load_compliance_plugins()
• Real-time monitoring: Background thread, alert deduplication
• Audit trail: Append-only SQLite log, JWT authentication, multi-tenancy
• Async jobs: SQLite-backed job queue with background worker
• Report export: PDF and CSV via /api/report endpoints

Operations

• REST API: Flask + Gunicorn — validation, anonymization, async jobs, report export, dataset registry, security endpoints, JWT auth
• Web dashboard: Dash + Bootstrap 5, multi-page admin UI (validate, dataset registry, jobs, custom compliance rules, audit log, users & tenants)
• CLI: per-column range/date/medical-code checks alongside file validation
• Docker: Multi-stage image, docker-compose with Redis

Quick Start

git clone https://github.com/RanaEhtashamAli/medical-data-validator.git
cd medical-data-validator

# Install with all extras
pip install -e ".[all,dev]"

# Run the REST API (port 8000)
python api.py

# Run the web dashboard (port 5000)
SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))") python launch_dashboard.py

Python Library

from medical_data_validator import MedicalDataValidator
from medical_data_validator.validators import PHIDetector, DataQualityChecker
import pandas as pd

validator = MedicalDataValidator(
    enable_compliance=True,
    compliance_template='clinical_trials',
)
validator.add_rule(PHIDetector())
validator.add_rule(DataQualityChecker())

data = pd.read_csv('your_medical_data.csv')
result = validator.validate(data)

print(f"Valid: {result.is_valid}")
print(f"Issues: {len(result.issues)}")

compliance = result.summary.get('compliance_report', {})
print(f"Overall Score: {compliance.get('overall_score', 'N/A')}")
print(f"Risk Level:    {compliance.get('risk_level', 'N/A')}")

FHIR R4 + SNOMED CT

from medical_data_validator.plugins import load_compliance_plugins

engine = load_compliance_plugins()          # auto-discovers installed plugins
report  = engine.comprehensive_compliance_validation(df)

Anonymization

anon_df = validator.anonymize(
    df,
    columns=['patient_name', 'ssn', 'email'],
    method='hipaa_safe_harbor',
)

CLI: per-column checks

medical-validator validate data.csv \
  --detect-phi --quality-checks \
  --range age:0:120 \
  --date-column visit_date --min-date 2020-01-01 --max-date 2026-12-31 \
  --code-column diagnosis_code:icd10 --code-column procedure_code:cpt \
  --format json --output results.json

--range/--date-column/--code-column are repeatable and can be combined with --profile. See medical-validator validate --help for the full flag list.

API Endpoints

/api/validate/data and /api/validate/file also accept optional validators (JSON-encoded per-column rules: required_columns, column_types, ranges, date_columns, min_date/max_date, code_columns), batch_size, and use_cache query params (or form fields for the file endpoint) — see API_DOCUMENTATION.md for the full schema.

JWT-protected endpoints use three roles (admin > data-steward > read-only); acquire a token via /api/auth/token and send Authorization: Bearer <token>. See API_DOCUMENTATION.md for request/response bodies, role requirements, and /v1.2/legacy-path equivalents.

Web Dashboard

/dash/ is a multi-page admin UI (Dash + Bootstrap):

Note: the dashboard calls the underlying business logic directly, in-process, as a fixed default-tenant identity — it does not go through the REST API's JWT/role checks. Treat /dash/ as an internal admin tool (e.g. behind a reverse-proxy auth layer or restricted network access), not as a role-scoped, multi-tenant-safe UI on its own.

Docker

cp env.example .env   # fill in SECRET_KEY, JWT_SECRET, ADMIN_PASSWORD
docker-compose up -d
# API: http://localhost:8000/api/health
# Dashboard: http://localhost:8000/home

Development

# Run all tests
pytest

# Skip slow/integration tests
pytest -m "not slow and not integration"

# Coverage report
pytest --cov=medical_data_validator --cov-report=term-missing

# Security scan
bandit -r medical_data_validator/ -ll

# Lint
flake8 medical_data_validator/
black medical_data_validator/
isort medical_data_validator/

Supported Standards

Configuration

Copy env.example to .env and set at minimum:

SECRET_KEY=<32-byte hex>
JWT_SECRET=<32-byte hex>
ADMIN_PASSWORD=<strong password>
FLASK_ENV=production

See docs/compliance_checklist.md for the full pre-production checklist.

Researcher Documentation

• docs/quickstart.ipynb — Jupyter notebook: validate → inspect compliance → anonymize → export reports → async jobs
• docs/compliance_checklist.md — HIPAA, GDPR, FDA 21 CFR Part 11, GCP feature mapping

License

MIT — see LICENSE.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions