pyfhircheck 0.2.0

Python FHIR R4 conformance validator with evidence and drift detection

pyfhircheck

Python FHIR R4 validator with evidence, drift detection, and CI-friendly output.

Validate FHIR JSON resources, Bundles, folders, or live server search results. pyfhircheck goes beyond JSON schema checks: it enforces structure, profiles, terminology, references, Bundle rules, and FHIRPath invariants, then writes reproducible evidence you can compare across runs.

The current official HL7 fhir-test-cases matrix run evaluates 165 supported R4 JSON cases: 152 have the same pass/fail result as the reference validator (92.1%), including 61/61 profile-module cases. These are selected-case agreement metrics, not a claim of complete validator parity. Validation evidence output inspired by MedVertical Records.

Why pyfhircheck

Deterministic reports	Every run gets a runId, issue fingerprints, config snapshot, and content hash
Evidence on disk	JSON report, OperationOutcome, CI summary, and manifest under evidence/<run-id>/
Drift detection	Compare two runs for new, resolved, and changed validation issues
Profile-aware	Load StructureDefinitions from local files, folders, .tgz packages, or remote URLs
Automation-ready	Machine-readable --agent-output, rule catalog, and structured logs to stderr

[!NOTE] pyfhircheck is a working validator with real validation depth covering structure, profiles, terminology, extensions, references, and Bundle semantics. See HL7 parity for current coverage and Current limitations for known gaps.

Features

Validation inputs

• Single resource file, Bundle, folder of JSON files, or resources fetched from a FHIR server
• Incremental folder validation with --changed-from (only re-validate changed files, keep reference context)

Structure and datatypes

• JSON validity, resourceType, cardinality, choice elements, unknown elements
• Primitive and complex datatype checks for common R4 clinical resources
• Contained resources, modifier extensions, and extension shape validation

Profiles and packages

• Enforced profiles from config and meta.profile
• FHIR NPM package resolution into a local cache (package-fetch)
• Snapshot + differential overlay for effective StructureDefinition elements
• Profile cardinality, fixed/pattern values, bindings, invariants, slicing (value/pattern/exists discriminators), and extension definitions

Terminology and references

• Local CodeSystem / ValueSet membership from packages (terminology.mode: off, local, strict)
• Reference resolution across contained resources, Bundle fullUrl, relative, absolute, and conditional references

Project rules and conformance

• Configurable custom rules (identifier systems, local reference resolution, Bundle resource types, and more)
• Conformance fixtures asserting PASS/WARN/FAIL plus expected issues or OperationOutcome-shaped expectations

Output

• Console summary, JSON report, OperationOutcome-compatible JSON, CI summary text
• --agent-output for a single JSON object with top issues, rule hints, and evidence path
• compare and export-evidence commands for drift workflows

Installation

Requirements: Python 3.11+

Install a published release from PyPI:

pip install pyfhircheck

Until the first PyPI release is published, install the current version directly from GitHub:

pip install "pyfhircheck @ git+https://github.com/N3RDMJ/pyfhircheck.git"

For development, clone the repository and use uv:

git clone https://github.com/N3RDMJ/pyfhircheck.git
cd pyfhircheck
uv sync

This creates .venv, installs locked dependencies from uv.lock, and installs pyfhircheck in editable mode with dev tools (pytest, mypy, build).

[!TIP] Without uv, use pip install -e ".[dev]" — the project stays compatible with standard PEP 517 tooling.

Build a wheel locally:

uv sync
uv run python -m build

Quick start

# Validate a resource
pyfhircheck file examples/valid-patient.json

# Validate with config (profiles, terminology, custom rules)
pyfhircheck file examples/valid-patient.json -c examples/pyfhircheck.json

# Fail CI on validation errors with structured outputs
pyfhircheck folder path/to/resources -c pyfhircheck.json \
  --json-output report.json \
  --ci-summary-output ci-summary.txt

Exit codes

Code	Meaning
0	Validation passed, or warnings below ciFailureThreshold
1	Validation failed (errors, or warnings when threshold is warning)
2	Config, evidence, package, or runtime error

CLI reference

Command	Description
file <path>	Validate one FHIR JSON resource
bundle <path>	Validate one Bundle resource
folder <path>	Validate all *.json files in a directory
scan <url>	Validate resources returned by configured searches (not a server conformance claim)
backend install|status|doctor	Manage the optional pinned HL7 Validator backend
validate-config	Check a config file without validating resources
package-fetch	Resolve configured FHIR packages into the local cache
conformance <path>	Run expected PASS/WARN/FAIL fixture cases
compare <before> <after>	Diff two evidence runs
export-evidence <run> <dest>	Copy an evidence run to another directory
rules	Print the machine-readable validation rule catalog
explain <code>	Explain a validation rule code

Common options (on file, bundle, folder, scan)

-c, --config PATH                  Validator config JSON
--json-output PATH                 Write full validation report JSON
--operation-outcome-output PATH    Write OperationOutcome-compatible JSON
--ci-summary-output PATH           Write one-line CI summary
--agent-output                     Single machine-readable JSON object on stdout
--max-issues N                     Limit issues in console/agent output
--fail-fast                        Show only the first issue
--changed-from RUN                 Validate only files changed since a prior evidence run
--log-level LEVEL                  Structured logs to stderr (DEBUG|INFO|WARNING|ERROR)
--backend native|hl7              Explicit validation backend (no fallback)

Examples

pyfhircheck file examples/invalid-patient.json --json-output report.json
pyfhircheck bundle examples/bundle.json -c examples/pyfhircheck.json
pyfhircheck scan https://hapi.fhir.org/baseR4 -c examples/pyfhircheck.json
pyfhircheck package-fetch -c examples/pyfhircheck.json
pyfhircheck conformance examples/conformance
pyfhircheck compare evidence/run-a evidence/run-b --fail-on-new-errors
pyfhircheck explain datatype.invalid --json

Python library

Validate resources returned by a live FHIR R4 API with bounded pagination, retry handling, and explicit scan-completeness metadata:

import os

from pyfhircheck import ApiScanOptions, scan_api

report = scan_api(
    "https://example.org/fhir",
    options=ApiScanOptions(
        resource_types=("Patient", "Observation"),
        search_params={"Patient": {"active": True}},
        auth=lambda: {"Authorization": f"Bearer {os.environ['FHIR_TOKEN']}"},
        timeout_seconds=20,
        max_pages=25,
        max_resources=2_000,
    ),
)

if not report.api_scan or not report.api_scan.complete:
    print(f"Incomplete scan: {report.api_scan.stop_reason if report.api_scan else 'unknown'}")

for issue in report.errors:
    print(issue.code, issue.path, issue.message)

When resource_types is omitted, pyfhircheck discovers the server's advertised resource types from /metadata. It accepts FHIR R4 servers, follows same-origin Bundle.link[relation=next] URLs, retries transient HTTP failures, and records redacted request diagnostics in report.api_scan. complete means only that configured searches were fully covered; it never asserts server conformance.

Run explicit API scenarios with typed builders or a versioned JSON pack:

from pyfhircheck import ApiTestOptions, load_builtin_api_test_plan, run_api_tests

api_report = run_api_tests(
    "https://example.org/fhir",
    load_builtin_api_test_plan("base-r4"),
    options=ApiTestOptions(
        inputs={"resourceType": "Patient", "resourceId": "p1", "versionId": "1"}
    ),
)
print(api_report.status, api_report.passed, api_report.failed, api_report.skipped)

Built-in packs are base-r4 and read-only isik3-basismodul. Missing identifiers produce SKIP, never PASS. Requests are read-only unless the scenario declares mutating=True and runtime execution also sets allow_mutation=True.

Validate an in-memory resource with the lower-level validator:

from pyfhircheck import Validator, ValidationReport
from pyfhircheck.config import ValidatorConfig

config = ValidatorConfig.load("pyfhircheck.json")
validator = Validator(config)

patient = {
    "resourceType": "Patient",
    "id": "example",
    "gender": "female",
}

report: ValidationReport = validator.validate_resource(patient)
print(report.status.value)          # PASS | WARN | FAIL
print(len(report.errors))           # error count
print(report.to_dict()["runId"])    # correlation / evidence id

Public exports also include ApiScanSummary, ApiScanStopReason, RequestDiagnostic, typed exceptions (ConfigError, PackageError, EvidenceError, …), and ValidationIssue.

Configuration

See examples/pyfhircheck.json for a working config.

Field	Purpose
fhirVersion	FHIR version (4.0.1 / R4)
backend	native (default) or explicit hl7
packages	NPM package id + version to resolve before validation
packageCacheDir	Local cache for resolved .tgz packages
localPackagePaths	Local StructureDefinition JSON, folders, or .tgz files
remotePackageSources	Remote .tgz package URLs
profiles	Enforced profile URLs per resource type
terminology.mode	off, local, or strict
terminology.endpoint	Explicit remote FHIR terminology base URL
terminology.bearerTokenEnv	Environment variable containing a bearer token
ciFailureThreshold	Fail CI on error (default) or warning
customRules	Project-specific rule settings
evidenceOutputDir	Where validation evidence is persisted
scanResourceTypes	Resource types covered by scan

Load config from a dict in code:

config = ValidatorConfig.load_dict({"fhirVersion": "4.0.1", "terminology": {"mode": "local"}})

Evidence and drift

Every validation run writes a directory under evidence/<run-id>/:

evidence/<run-id>/
├── manifest.json           # run metadata and file index
├── report.json             # full validation report
├── operation-outcome.json  # OperationOutcome-compatible issues
├── ci-summary.txt          # one-line PASS/FAIL summary
├── config.json             # config snapshot used for the run
└── inputs.json             # input file content hashes

Compare two runs:

pyfhircheck compare evidence/run-a evidence/run-b --fail-on-new-errors

The diff reports new errors, resolved issues, severity changes, and config/profile/terminology drift.

CI integration

pyfhircheck folder fhir-resources -c pyfhircheck.json \
  --json-output validation-report.json \
  --operation-outcome-output operation-outcome.json \
  --ci-summary-output validation-summary.txt

Use exit code 1 to fail the pipeline when validation errors are present. Set "ciFailureThreshold": "warning" in config if warnings should also fail CI.

[!TIP] Pair --changed-from with evidence from a previous run to validate only modified files while keeping unchanged resources available for reference resolution.

Agent and automation output

For LLM agents and CI parsers, use --agent-output to emit a single JSON object (pyfhircheck.agent-output.v1) with status, truncated top issues (including rule hints and fingerprints), and the evidence path.

pyfhircheck file patient.json --agent-output --max-issues 5
pyfhircheck rules   # machine-readable rule catalog
pyfhircheck explain profile.required --json

Observability

Structured JSON logs are written to stderr (stdout stays clean for --agent-output and piped JSON).

export PYFHIRCHECK_LOG_LEVEL=INFO      # default: WARNING
export PYFHIRCHECK_LOG_FORMAT=json   # or console

pyfhircheck file patient.json --log-level INFO

Logs include correlation IDs, run timing, package download retries, and validation summaries.

Conformance fixtures

Fixture files assert expected validation outcomes. Minimal case:

{
  "expectedStatus": "PASS",
  "resource": {"resourceType": "Patient", "id": "p1", "gender": "female"}
}

Issue-level expectations:

{
  "expectedStatus": "FAIL",
  "expectedIssues": [
    {"severity": "error", "code": "datatype.invalid", "path": "Patient.id"}
  ],
  "resource": {"resourceType": "Patient", "id": "bad id"}
}

expectedOperationOutcome.issue is also accepted for OperationOutcome-compatible matching. Run fixtures with:

pyfhircheck conformance examples/conformance

Development

uv sync
uv run pytest tests/ -v
uv run mypy src/pyfhircheck/
uv run python -m build

GitHub Actions CI runs lint, type check, tests, and build on every push and PR. See .github/workflows/ci.yml.

To publish a release, configure PyPI trusted publishing for the pypi GitHub environment, update pyfhircheck.__version__, and publish a GitHub release tagged with either the version (for example 0.2.0) or a v prefix (v0.2.0). The publish workflow verifies the tag, runs the test suite, builds the distributions, and publishes them without a long-lived API token.

HL7 parity

pyfhircheck is tested against the official HL7 fhir-test-cases conformance suite. Each test case includes a FHIR resource and the expected HAPI validator outcome.

Metric	Value
Manifest cases recorded	972
Supported R4 JSON cases evaluated	165
Matches (same pass/fail as the reference validator)	152
Mismatches	13
Explicitly skipped with a reason	807
Profile-module matches	61/61
Selected-case pass/fail agreement	92.1%

Run the parity suite yourself:

from pathlib import Path
from pyfhircheck.parity.hl7_runner import run_hl7_test_cases, format_hl7_report

report = run_hl7_test_cases(Path("/tmp/fhir-test-cases"), Path(".pyfhircheck/packages"))
print(format_hl7_report(report))

The percentage is selected-case pass/fail agreement, not feature coverage or full validator parity. Coverage matrices keep every manifest case visible as evaluated or skipped with a reason and also compare normalized severity, category, and location.

Optional HL7 backend

The native R4 backend remains the default. Install the pinned, checksum-verified 6.10.0 Java artifact explicitly before selecting it:

pyfhircheck backend install
pyfhircheck backend doctor
pyfhircheck file patient.json --backend hl7

Validation never downloads the JAR and never falls back between backends.

Current limitations

[!WARNING] Native validation remains R4 JSON only. Differential snapshot, slicing/reslicing, FHIRPath, narrative, and terminology semantics continue to be measured against the pinned HL7 backend; XML resource input and R4B/R5 native engines are out of scope.

Roadmap

See docs/parity-roadmap.md for parity gates and current status against HAPI / HL7 validator behavior.