fhirsync

fhirsync copies FHIR resources from a source FHIR server or from FHIR files to a target FHIR server or to files. It searches the source server (or reads the files), follows the references of every result and puts each result together with the resources it references into a FHIR Bundle, so every Bundle can be written to the target on its own. On the way, configurable processing rules filter and change the resources, for example to point references to the target's patients or to give resources stable identifiers.

fhirsync is a command line tool and a Python library. Its main use today is syncing patient consents from gICS (the consent management of the THS Greifswald) to a FHIR server: on demand with the fhirsync command, and event-driven with the separate project gics-consent-sync, which listens to gICS's MQTT notifications and runs fhirsync as a library. Nothing in fhirsync is specific to gICS, though: what happens to the resources is entirely defined by the configuration.

• How it works
• Requirements
• Installation
• Usage
• Configuration
• Processing rules
• Using fhirsync as a library
• Important behaviour
• Syncing gICS consents
• Upgrading from fhirsync 0.8 and 0.9
• Upgrading from fhirsync 0.7 and older
• Testing
• Project structure, and ARCHITECTURE.md for developers

How it works

flowchart LR
    source[("source: FHIR server<br/>e.g. gICS TTP-FHIR Gateway,<br/>or FHIR files")]
    subgraph fhirsync
        direction TB
        filter["1. filter rules"] --> references["2. follow references<br/>(recursive)"]
        references --> transform["3. transform rules"]
        transform --> request["4. build Bundle entry requests"]
        request --> finalize["5. finalize rules"]
    end
    target[("target<br/>FHIR server or JSON files")]

    source -- "search results<br/>or the resources of the files" --> filter
    references -- "read referenced resources" --> source
    finalize -- "transaction Bundles<br/>PUT / POST" --> target

For every result (a search result, or a resource from the files) fhirsync:

1. Filters it: rules can skip resources or read values from them.
2. Follows its references: resources referenced as Type/id are read from the source and processed the same way, recursively. Every Reference counts, wherever it is in the resource, also in lists like Observation.performer. References to other resources in the Bundle are rewritten to the conditional form Type?identifier=system|value, so the target links them to its own copies.
3. Transforms it: rules set and remove elements and identifiers.
4. Builds the request of its Bundle entry, by default a conditional update PUT Type?identifier=system|value.
5. Finalizes it: rules remove what only the request needed, e.g. the source server's id.

A Bundle holds a configurable number of results with their referenced resources. It is posted to the target server, or written to a file.

The source is one of two types:

• server: a FHIR server. fhirsync runs the searches configured in source.searches, or the search given with -q, one after another, and syncs their results.
• files: FHIR files in a directory (JSON, NDJSON or FHIR XML, optionally gzip compressed). All files are read, every resource in them is a result (or only those of source.resource_types), and references are resolved from the files.

The fhirsync command does one such run, for initial loads, re-syncs, imports or tests. Applications run the same from their own code (see Using fhirsync as a library).

Requirements

• Python 3.13 or newer and uv.
• A source FHIR server reachable from the host, or FHIR files. For gICS, see Syncing gICS consents.

Installation

git clone <repository url> fhirsync
cd fhirsync
uv sync

All dependencies come from PyPI, including fhirsync's HTTP and file helpers dic-helper-toolkit. fhirsync itself is published on PyPI as fhir-sync; applications add it as a dependency (see Using fhirsync as a library).

uv sync also installs the development tools (pytest, responses) from the dev dependency group. For a production installation without them, use uv sync --no-dev.

Reading FHIR XML files needs the extra xml (it installs lxml through dic-helper-toolkit[xml]): uv sync --extra xml, or fhir-sync[xml] as a dependency.

Usage

uv run fhirsync [-c CONFIG] [-q QUERY]

Option	Meaning
-c, --config	the configuration file, default: config.yaml in the current directory
-q, --query	a FHIR search instead of source.searches, e.g. Consent?domain:identifier=MII; source type server only

Searches (source type server)

Without -q, fhirsync runs the searches of source.searches, one after another:

source:
  base_url: https://gicsserver/ttp-fhir/fhir/gics/
  # what applies to every search of a type
  request_params:
    Consent:
      category: http://fhir.de/ConsentManagement/CodeSystem/ResultType|consent-status
  # what to fetch: only what is specific to each search
  searches:
    - Consent?domain:identifier=MII
    - ResearchStudy

-q replaces them for one run, e.g. for a re-sync or a single patient:

# all consent-status Consents of the MII domain (the category comes from source.request_params)
uv run fhirsync -c config.yaml -q 'Consent?domain:identifier=MII'

# all resources of a type, with the parameters configured for it
uv run fhirsync -c config.yaml -q Consent

# a single patient's consent status
uv run fhirsync -c config.yaml -q 'Consent?domain:identifier=MII&patient.identifier=https://ths-greifswald.de/fhir/gics/identifiers/Pseudonym|1000000001'

• A search is <ResourceType> or <ResourceType>?<parameters>, relative to source.base_url. Quote -q in the shell, because of & and |. -q takes one search.
• searches and request_params have different roles. A search says what is fetched: put only what is specific to it into the search, e.g. a domain or a patient. source.request_params says how every search of a resource type is made: put what applies to all of them there, e.g. _count, _profile or a category. They are added to every search of that type: the searches, -q, searches of applications such as gics-consent-sync, and searches for Type?… references. A parameter given in both places takes the search's value, except for Type?… references, where the configured value wins.
• Paging: fhirsync pages through all results, following next links, or using offsets if source.next_param is set.
• Without -q and without source.searches there is nothing to run, which is an invalid command line.

Files (source type files)

source:
  type: files
  directory: /data/export
  recursive: true
  resource_types: [Consent]

uv run fhirsync -c config.yaml

• All files are read first, when fhirsync starts: the files in directory (and its subdirectories with recursive) whose names match patterns, by default all JSON, NDJSON and XML files, also gzip compressed. Hidden files are skipped. A missing directory is an error.
• Formats: .json (a resource), .ndjson (a resource per line), .xml (a resource in FHIR XML, needs the extra xml), each optionally .gz. Bundles in files give the resources of their entries (unpack_bundles).
• Results: every resource of the files, or only those of resource_types; the others are then only used when they are referenced. Each file is one page: with bundling.bundle.size: 0, one Bundle per file.
• References are resolved from the files, by type and id. A resource that occurs twice is used once, with a warning. References written as Type?<parameters> can't be resolved from files (see Important behaviour).
• Invalid files stop the run (on_error: raise), or are skipped with a warning (on_error: skip).
• -q can't be used with files.

Exit codes and logging

• 0 means success.
• 2 means an invalid command line or configuration, e.g. no search for a server source, or -q with a files source. The configuration error is logged with all problems found.
• 1 means an error while running, e.g. an unreachable server, an invalid file, a failing rule, a missing reference, or a resource without the identifier its request needs (see Important behaviour). The error is printed with a traceback.

Logging goes to standard error, with timestamps, at level INFO. Each processed resource, fetched reference and written Bundle is logged. At the end, the number of written and rejected Bundles is logged.

Configuration

The configuration is a YAML file. config-example.yaml documents every setting with its default. Copy it to config.yaml and adapt it. When fhirsync starts, it validates the whole file and reports all problems at once:

Invalid configuration in config.yaml:
  source.auth: auth_url, realm, client_id also need client_secret
  bundling.bundle.sise: unknown configuration variable, did you mean 'size'?

• Errors stop fhirsync before anything is fetched or sent: unknown variables (with a suggestion for typos), incomplete settings (half an authentication, a target mode without its URL or directory), invalid values, and invalid rules.
• Warnings are logged for settings that have no effect, e.g. target.outdir: ignored, not used in mode server. fhirsync continues.

Secrets and environment variables

Any text value can use ${NAME} or ${NAME:-default}:

source:
  auth:
    auth_url: https://authserver/auth
    realm: REALM
    client_id: fhirsync
    client_secret: ${GICS_CLIENT_SECRET}
target:
  mode: server
  base_url: ${FHIR_TARGET_URL:-https://fhirserver/fhir}

• An unset variable without a default is an error that names the variable and the setting.
• Substitution happens after the YAML is read, so values may contain any characters.
• Inside inline mappings ({ … }), quote values that contain ${…}.

Settings reference

Paths are written as section.key. "common" in a per-type section applies to all resource types, and a type's own setting overrides it (see below).

common: defaults for the TLS connections to source and target

Setting	Default	Meaning
common.ca_file	none (system CAs)	CA certificate file (PEM). If set, only this file is trusted.
common.rfc5280	true	strict RFC 5280 certificate checks; false for non-conforming internal CAs

source: where the resources come from (required)

Setting	Default	Meaning
source.type	server	server: a FHIR server; files: FHIR files. The settings of the other type are errors.

With type: server:

Setting	Default	Meaning
source.base_url	required	base URL of the FHIR endpoint
source.auth.username, .password	none	basic authentication; both or neither
source.auth.auth_url, .realm, .client_id, .client_secret	none	OAuth 2.0 via Keycloak; all four or none. With username/password: password flow, otherwise client credentials flow
source.ca_file, source.rfc5280	common.*	TLS settings for this server
source.next_param	none	query parameter for offset paging, for servers without next links (gICS), e.g. _offset
source.request_params.<common|Type>.<param>	none	defaults for every search of the type (searches, -q, applications, Type?… references); a search's own parameters take precedence, except in Type?… references; a list value repeats the parameter
source.searches	none	the searches of a run, each <Type> or <Type>?<parameters>; -q replaces them. Unknown resource types are errors.

With type: files:

Setting	Default	Meaning
source.directory	required	the directory with the FHIR files
source.patterns	*.json, *.ndjson, *.xml, *.json.gz, *.ndjson.gz, *.xml.gz	file name patterns (*, ?, [seq], case-sensitive) of the files to read
source.recursive	false	also read the files of all subdirectories
source.include_hidden	false	also read hidden files (.name) and hidden subdirectories
source.follow_symlinks	false	also read subdirectories that are symbolic links
source.unpack_bundles	true	a Bundle in a file gives the resources of its entries; false: the Bundle itself
source.on_error	raise	an invalid file, line or resource: raise stops the run, skip logs a warning and continues
source.resource_types	all	the resource types that are results; the others are only used as reference targets. Unknown resource types are errors.

target: where the Bundles go (required)

Setting	Default	Meaning
target.mode	required	server or file
target.base_url	required with server	base URL of the target FHIR endpoint
target.outdir	required with file	directory for the Bundle files <identifier>.json
target.auth.*	none	as source.auth, only with server
target.ca_file, target.rfc5280	common.*	TLS settings, only with server

bundling: Bundles and requests

Setting	Default	Meaning
bundling.bundle.type	transaction	transaction, batch or collection
bundling.bundle.size	1	results per Bundle, plus their referenced resources; 0 for one Bundle per search page or file
bundling.bundle.identifier_system	http://www.uniklinikum-jena.de/fhir/sid/consentBundle	system of Bundle.identifier; its value is a random UUID
bundling.resources.<common|Type>.request.method	put	put (create or update) or post (create)
bundling.resources.<common|Type>.request.url	identifier	for put: identifier (PUT Type?identifier=…, needs an identifier) or id (PUT Type/<source id>)
bundling.resources.<common|Type>.request.condition	none	for post: ifNoneExist, create only if no resource with the identifier that counts exists
bundling.resources.<common|Type>.reference_mode	identifier	identifier: rewrite references from this type to resources in the Bundle to Type?identifier=…; as_is: leave them
bundling.resources.<common|Type>.ignore_references_to	[]	resource types whose references from this type are not followed (and not rewritten)
bundling.resources.<common|Type>.on_missing_reference	error	a reference from this type the source doesn't have: error stops the run, warning logs it and leaves the reference unchanged
bundling.resources.<common|Type>.identifier_system	none	the system of the identifier that counts for this type's requests, ifNoneExist conditions and references to it; none: the first identifier
bundling.resources.<common|Type>.on_missing_identifier_system	error	a resource without an identifier of identifier_system: error stops the run, first logs a warning and uses its first identifier. Only with identifier_system.

processing: filtering and changing resources, see docs/rules.md

Setting	Default	Meaning
processing.identifier_systems.<Type>	none	identifier system per resource type, for identifierSystem('<Type>') in rules
processing.rules.<Type|"*">.<filter|transform|finalize>	none	the rules

common and per-type settings

In source.request_params and bundling.resources, the key common applies to all resource types, and a resource type's own key overrides it:

• Single values (method, url, condition, reference_mode, on_missing_reference, identifier_system, on_missing_identifier_system) are overridden one by one. With common.request: {method: put, url: id} and Patient.request: {url: identifier}, Patient gets put with identifier.
• Lists (ignore_references_to) are replaced as a whole. Encounter: {ignore_references_to: []} follows all references, even if common ignores some.
• Search parameters are merged parameter by parameter. With common: {_count: '10', _lastUpdated: ge2025} and Consent: {_lastUpdated: ge2026}, Consent searches use _count=10&_lastUpdated=ge2026.

Processing rules

Rules filter and change the resources. They are written per resource type and phase, and use FHIRPath to read values:

processing:
  identifier_systems:
    Patient: http://hospital.example/sid/patient
  rules:
    Consent:
      filter:
        - skip: {when: "status != 'active'"}
        - extract: {name: pat_id, from: patient.display, pattern: '^\s*\S+\s+(\d{10})\s*$'}
      transform:
        - set: {path: patient.reference, value: "Patient?identifier={identifierSystem('Patient')}|{%pat_id}"}
        - delete: patient.display
    "*":
      finalize:
        - delete: [id, meta.lastUpdated]

docs/rules.md describes the phases, every operation, the expression language and its pitfalls, with a cookbook of tested real world examples. Examples include skipping resources, stable identifiers from natural keys, pseudonymisation, tagging, and custom Python functions.

Using fhirsync as a library

Applications add fhir-sync as a dependency and import the package fhirsync:

from fhirsync import ConfigError, ResourceBundler, load_config

config = load_config("config.yaml")              # raises ConfigError with all problems
bundler = ResourceBundler(config)                # one bundler can run many searches
result = bundler.fetch_resources("Consent", params={
    "patient.identifier": "https://ths-greifswald.de/fhir/gics/identifiers/Pseudonym|1000000001",
})
print(result.bundles, result.entries, result.rejected)

API	Meaning
load_config(path)	loads and validates a fhirsync configuration file, as the command does; raises ConfigError
ResourceBundler(config)	the bundler for a configuration; for a files source, all files are read here
ResourceBundler.run(searches=None)	runs like the command: the searches of a server source (searches instead of source.searches, like -q), or the files of a files source; returns the added-up SyncResult
ResourceBundler.fetch_resources(search, params=None)	runs one search of a server source and returns a SyncResult
SyncResult	bundles and entries written to files or accepted by the target, rejected: Bundles the target answered with 404
load_yaml(path, model), ConfigModel	load an application's own YAML configuration like fhirsync's (see below)
Config, ConfigError, RuleError, MissingReferenceError	the configuration model and the errors

• Search parameters in params are not URL encoded; a list value repeats the parameter. They take precedence over the parameters in search, which take precedence over source.request_params. As for searches: pass what is specific to this search, and leave what applies to every search of the type to source.request_params.
• Errors are raised, as with the command: RuleError for a failing rule, MissingReferenceError for a missing reference with on_missing_reference: error, ValueError for a resource without the identifier its request needs and for searches with a files source, requests exceptions for connection and HTTP errors, and dic-helper-toolkit's FHIRFileError for an invalid file with on_error: raise. A 404 from the target is only counted in rejected.
• Logging uses the loggers fhirsync.* and is not configured by fhirsync; that is up to the application.
• Own configuration files: a class derived from ConfigModel and loaded with load_yaml(path, Model) gets the same behaviour as fhirsync's file: ${…} environment variables, all errors at once with suggestions for typos, and warnings for the settings its ignored_settings() reports.

Important behaviour

Without rules, no resource is changed. Resources are still fetched, bundled and sent, but exactly as the source delivered them, including the source server's id and meta.lastUpdated. Many servers reject conditional updates whose body contains a different id. Almost every configuration therefore needs at least:

processing:
  rules:
    "*":
      finalize:
        - delete: [id, meta.lastUpdated]

A resource without an identifier stops the whole run when its request is identifier based (request.url: identifier, the default, or condition: ifNoneExist). The error is Resource <Type>/<id> has no identifier.. This is intended: no resource is written to the target without a stable identity. Resources whose source has no identifier need a set_identifier rule; gICS's consent-status Consents are an example. Otherwise use request.url: id for that type.

The first identifier counts, unless identifier_system is configured for the resource type. Request URLs, ifNoneExist conditions and rewritten references all use a resource's first identifier, or the one with the configured system. A resource without an identifier of that system stops the run, unless on_missing_identifier_system: first is set. This also applies to resource types with a single identifier, such as QuestionnaireResponse.

A failing rule stops the run, e.g. a template expression without a result. Invalid rules are already reported at startup.

A missing reference stops the run by default (on_missing_reference: error): a referenced Type/id the source server answers with 404, or that the files don't contain, and a Type?<parameters> reference whose search finds nothing (with files: every such reference). The error names the reference, the resource and the setting. With on_missing_reference: warning for the referencing type, the reference is logged and left unchanged.

• Ignored references are not rewritten. A reference to a type in ignore_references_to keeps pointing to the source server's id. Set it with a rule if the target needs it, as the gICS rules do for Consent.patient.
• Skipping leaves references unchanged. If a filter rule skips a referenced resource, the reference to it is not rewritten.
• Duplicates: a resource that several results reference is only added once per Bundle.
• Empty Bundles, e.g. when all resources were skipped, are not written.

Syncing gICS consents

config-example.yaml is a complete setup for syncing the consent-status Consents of a gICS domain MII. Its rules reproduce the processing that was built into fhirsync 0.7 and older (see docs/rules.md, section 7.2).

The separate project gics-consent-sync syncs consents event-driven, from gICS's MQTT notifications. Its README also describes what this setup expects from gICS, the tested gICS versions, and how to convert configurations of fhirsync 0.7 and older.

Upgrading from fhirsync 0.8 and 0.9

fhirsync 0.10 adds FHIR files as a source (source.type: files) and searches in the configuration (source.searches). Existing configurations stay valid: source.type defaults to server.

• -q is optional: without it, the searches of source.searches are run. A server source without both is an invalid command line (exit code 2) instead of the former "required argument" error.
• Missing references stop the run by default (on_missing_reference: error). A Type/id reference the server answers with 404 already stopped the run, now with a MissingReferenceError instead of an HTTPError. New: a Type?<parameters> reference whose search finds nothing stops the run too; before, it was silently left unchanged. Set bundling.resources.<common|Type>.on_missing_reference: warning for the previous behaviour.
• References in lists are followed: References held directly in a list, e.g. Observation.performer or Consent.organization, were not found before, so they were neither followed nor rewritten. Now they are, like every other reference; a missing one stops the run by default (see above). The gICS rules are not affected: gICS Consents only carry the organization's name.
• Choosing the identifier: bundling.resources.<common|Type>.identifier_system picks the identifier with that system for requests and references, instead of the first one. Without it nothing changes.
• Library: ResourceBundler.run() runs like the command; fetch_resources() is still there for single searches of a server source. ResourceBundler.source_client is None for a files source.

Upgrading from fhirsync 0.7 and older

fhirsync 0.8 replaces the INI configuration with YAML, and the processing that was built in is now written as rules. Sync mode (fhirsync -s) moved to the separate project gics-consent-sync, which uses its own configuration file for the MQTT settings. Its converter turns an INI configuration into both files:

# in gics-consent-sync
uv run python tools/convert_ini_config.py config.ini config.yaml gics-consent-sync.yaml

Without sync mode, only config.yaml is needed. Other changes for query mode:

• Strict RFC 5280 certificate checks are on by default; set rfc5280: false for non-conforming internal CAs.
• Offset paging is only used when next_param is set.
• ifNoneExist conditions use the FHIR standard form identifier=system|value.
• Common and type-specific search parameters are merged parameter by parameter. Before, the type-specific ones replaced the common ones completely.
• Code that imported the package sync imports fhirsync now.

Testing

uv run pytest tests

Each test checks one documented behaviour. A comment above it names the section of this README, docs/rules.md or config-example.yaml it verifies.

• tests/config/: loading and validating configurations, defaults, errors, warnings, environment variables
• tests/rules/: the rule engine: operations, templates, FHIRPath, variables, startup checks
• tests/cookbook/: the cookbook examples of docs/rules.md
• tests/bundling/: the bundler with rules, references, identifiers and requests, on recorded gICS responses
• tests/sources/: the source types: files, the searches of a server source, missing references
• tests/library/: the library API
• tests/cli/: the fhirsync command: options, exit codes, logging
• tests/connections/: authentication and TLS of the source and target connections
• tests/project/: project structure, Python version and dependencies

The recorded gICS responses in tests/data/ come from gics-consent-sync's real world data tests.

Project structure

fhirsync/
  __init__.py      the library API
  cli.py           the fhirsync command
  config.py        configuration: YAML loading, validation, environment variables
  bundler.py       results, references, Bundles, requests, output
  sources.py       the source types: FHIR server and FHIR files
  rules.py         the processing rule engine
docs/
  rules.md         processing rules and expression language
tests/
  config/ rules/ cookbook/ bundling/        tests, see Testing
  sources/ library/ cli/ connections/ project/
  support.py, conftest.py                   shared test functions and fixtures
  data/                   resources, rules and recorded responses used by the tests
ci/
  verify_published_package.py   CI check of the package published on PyPI
config-example.yaml       documented example configuration
ARCHITECTURE.md           how fhirsync is built, for developers
.github/workflows/ci.yml  CI (GitHub Actions): build, test, publish to PyPI, tag the release