dirigent-integration

The control center for the dirigent plugin ecosystem. This is the one place aware of every official pack at once -- it assembles them into a single environment and runs everyone together to prove they still compose.

The principle it protects

dirigent-core knows nothing about any pack. Every adapter, DHIS2 today and more to come -- anything the core is deliberately ignorant of -- lives only in its own repository and self-tests there, against nothing but its own dependencies. The integration is the only place the whole set is installed side by side, so it is the only place that can prove they still compose: that every pack's blocks land in one catalog without colliding, that every pack's examples still validate against that merged catalog, and that a pipeline spanning two packs is runnable at all.

What is here

• ecosystem.yaml -- the manifest: every component and the ref it tracks, the source of truth for what gets assembled. dirigent (the runtime monorepo) is role: runtime, installed as dependencies and never cloned; each adapter is role: pack, cloned and self-tested here. dirigent-dhis2 is the first; a future pack becomes real with a single entry.
• pyproject.toml -- the assembled runtime: every dirigent runtime package plus every pack, wired through [tool.uv.sources] as git dependencies on branch = main, unpinned, so the integration always assembles the latest ecosystem.
• scripts/run_integration.py -- the runner. It reads the manifest, clones each pack, and runs everything against the assembled environment. It is generic: it iterates the packs the manifest names and has no per-pack knowledge.
• infra/ -- the batteries-included image and the stack that runs it: Dockerfile builds dirigent plus every pack plus the web UI, compose.yaml runs it against postgres.
• src/dirigent_integration/shelves/ -- cross-boundary example pipelines, authored here because they span more than one pack and belong to none. They ship as package data: this repository is itself a distribution registering one plugin whose examples() hook carries the shelves, so an instance with it installed lists them with dg examples list --plugin integration. The root examples/ is a symlink to them, so a checkout's paths still read.
• tests/ -- the integration's own tests: the merged catalog is coherent, and the cross-boundary examples validate against it.

Running it locally

make test

One target, one green or red:

1. clone every role: pack in ecosystem.yaml into checkouts/ (gitignored), at its ref;
2. run each pack's own pytest suite against the assembled environment;
3. validate each pack's own examples against the merged catalog;
4. validate the cross-boundary examples against the merged catalog;
5. run the integration's own tests;
6. ask every DHIS2 play demo an example names for its system info, and fail on any that no longer answers;
7. check those demos against the DHIS2 Instance Manager's public list, and fail on one whose version has a newer stable patch, before the old address is retired;
8. rehearse an export and a dry-run import against the newest stable demo of 2.42 and of 2.43, the versions the packs target, and fail unless values moved and the import was clean.

A pack whose checkout is already present is moved to its ref's tip, so the tests read from the same commit the source was installed from; one with local changes is left where it is and reported, which is what lets a copy of a sibling working tree stand in for a clone. Other targets: make lock, make sync, make clone, make lint, make clean.

An instance to look at

make dev          # an empty instance on http://127.0.0.1:3333
make dev-seeded   # the same instance with every corpus in it

make dev-seeded names no directory at all: every corpus is installed -- dirigent's own as dirigent-examples, each pack's inside the pack, and this repository's inside dirigent_integration -- and each ships through the examples() plugin hook, so dg dev --seed-installed reaches all of them. A pack added to ecosystem.yaml is seeded by being installed. Schedules land paused, and a document the instance will not store is reported and passed over -- a corpus holds those on purpose, and a refusal is itself something to look at.

The runtime is installed from git rather than cloned, so make clone also checks out checkouts/dirigent, which exists for one thing: make ui builds the web bundle there. make test neither needs nor makes that checkout. DEV_HOST and DEV_PORT move where both targets listen.

dev-seeded mints a DIRIGENT_SECRET_KEY per boot, because a connection carrying a credential cannot be stored without one and the state is wiped first anyway, and it allows the unsafe blocks the corpora teach with (UNSAFE_BLOCKS): shell.run, and the docker blocks that run, build and compose. What stays refused is what no local instance holds: a connection to a system nothing here runs, such as an object store, a warehouse or ODK Central, that no file in the corpora declares.

dirigent-server installed from git carries no UI bundle -- its static/ is gitignored there -- so both targets build one in the cloned checkout (make ui, which needs bun) and point the server at it with DIRIGENT_UI_DIR. Without bun the build is skipped with a note and the instance answers the API alone; the stack make up runs always has the UI.

The cross-boundary examples

Pipelines that no single pack owns because they span several. Today they exercise the first pack, DHIS2, across the built-in storage and transform packs; as more adapters land, cross-boundary pipelines across them are authored here the same way.

• examples/dhis2-export-to-s3.yaml -- a dhis2.data_value_set_export written to the s3:// scheme with storage.write, then a storage.copy archiving it.
• examples/dhis2-values-to-parquet.yaml -- the same export staged in s3:// and fed to convert.arrow, re-encoding to parquet beside it: three packs in one pipeline.
• examples/dhis2-values-per-org-unit-to-parquet.yaml -- that export fanned out over a list of organisation units, flattened into one parquet table, with a manifest of what landed.
• examples/dhis2-analytics-to-csv-report.yaml -- a dhis2.analytics_query grid flattened into rows, written to s3:// as csv, and announced with a webhook.post summary.
• examples/parquet-to-dhis2-import.yaml -- the return leg: a parquet drop waited on with storage.exists, decoded, gated on a carried schema, then imported into DHIS2.
• examples/dhis2-tracker-weekly-window.yaml -- a windowed weekly schedule driving dhis2.tracker over the week that just closed, archived as ndjson and parquet in s3://.
• examples/dhis2-metadata-snapshot.yaml -- a metadata snapshot written to s3://, then pipeline.run of dhis2-export-to-s3: composition across the corpus itself.

FHIR examples

examples/fhir/ is a shelf of its own: nine pipelines moving data between a FHIR endpoint and DHIS2, in both directions. They are cross-boundary twice over -- the DHIS2 blocks come from the dirigent-dhis2 pack and the HTTP, transform and validate blocks from the runtime, and neither knows the other exists.

Most of them read a d2w fhir serve facade, which publishes one DHIS2 instance as a FHIR endpoint and takes captures back; where a general-purpose FHIR server is the point -- an Encounter, an Observation, a Subscription, a _lastUpdated search, none of which that facade serves -- the default is a public HAPI R4 sandbox instead, and the document's header says so. Every write is a dry run: dryRun=true on /api/dataValueSets, importMode=VALIDATE on /api/tracker.

examples/fhir/README.md has the one-line table and the mapping vocabulary the shelf is built on -- subject, capture, ConceptMap.

• examples/fhir/fhir-capture-bundle-to-data-values.yaml -- the capture pair end to end: a page of captures pulled off the facade, translated, gated on the pack's own id and period formats, rehearsed and imported.
• examples/fhir/fhir-questionnaire-response-to-data-values.yaml -- the general case, where a linkId is a form designer's name and the lookup is the integration.
• examples/fhir/fhir-patient-to-tracked-entity.yaml -- the Patient register folded into an /api/tracker registration, posted by http.request because dhis2.tracker only reads.
• examples/fhir/fhir-encounter-to-event.yaml -- one Encounter and its Observations folded into the single program stage event DHIS2 wants.
• examples/fhir/fhir-conceptmap-driven-mapping.yaml -- the mapping fetched from the server at run time rather than written into the document.
• examples/fhir/fhir-measure-report-to-analytics-check.yaml -- a MeasureReport held against dhis2.analytics_query for the same period, and the gap delivered by webhook.post.
• examples/fhir/dhis2-to-fhir-observations.yaml -- the outbound leg: a data value set published as a transaction Bundle of conditionally-updated Observations.
• examples/fhir/fhir-subscription-webhook-to-dhis2.yaml -- the subscription contract, landing on a webhook whose payload mapping no caller can reach past.
• examples/fhir/fhir-nightly-window-sync.yaml -- a half-open _lastUpdated window, a fan-out per resource type under items: continue, and a join that says what it missed.

Inbound examples

examples/inbound/ is a shelf of eight pipelines going one direction -- an outside system, a transform, DHIS2 -- teaching the shapes an inbound integration actually takes: a public JSON API, a csv dropped in a bucket, a pushed webhook, a parquet table from a lakehouse, a FHIR Observation feed, a windowed weekly pull, a fan-out across facilities, and the defensive version with every guard on. Each names every block it needs, carries its DHIS2 connection pointed at the public play server, and imports as a dry run by default, so dg run --local examples/inbound/<file> works with nothing set up. examples/inbound/README.md lists them one line each.

The batteries-included image

dirigent-full is dirigent with every official pack the manifest names, plus the web UI, in one image, built here with make image and not published. The published image is the core one, ghcr.io/winterop-com/dirigent, which carries no adapter: a deployment that wants a pack builds one layer on it (dirigent's operations guide, "The image"), and dg init --template compose writes that Dockerfile. This repository builds the assembled image because it is the one place the whole set is already resolved, and the stack below runs it.

infra/Dockerfile builds it from this repository's pyproject.toml and uv.lock in four stages: a lock stage where scripts/dirigent_rev.py reads the dirigent commit out of the lock, a bun stage that builds the UI bundle from that very commit (dirigent-server installed from git carries no bundle, because its static/ is gitignored there), a uv stage that runs uv sync --locked --no-dev, and a runtime stage that mirrors the runtime stage of dirigent's own infra/Dockerfile and must move with it.

The build clones the components over public HTTPS and needs no credentials. A component repository that is private needs a GitHub read token instead; it goes in as a BuildKit secret and is exposed to git only inside the one RUN that needs it, so it never lands in a layer or a config file.

make image                             # public components, no token
export GITHUB_TOKEN=$(gh auth token)   # only if a component repository is private

Running the stack

cp .env.example .env      # set DIRIGENT_SECRET_KEY and DIRIGENT_BOOTSTRAP_ADMIN_PASSWORD
make up

Four services -- postgres, a one-shot migration, the server and a worker -- with the cross-boundary shelves mounted as the apply directory, so those pipelines are seeded at boot. The UI is on http://localhost:3333; make down removes it, volumes and all.

CI

The assembled check is expensive -- clone every pack, install the whole runtime, run every suite -- so it is deliberately infrequent. Per-commit safety belongs to each repository's own CI: dirigent tests itself on its pushes, each pack tests itself on its own. The integration is the integration truth, caught once a day. .github/workflows/ci.yaml therefore triggers only on schedule: (a nightly cron) and workflow_dispatch: (manual); it never runs on push, which would turn one commit into a full-ecosystem rebuild. A lightweight .github/workflows/lint.yaml runs on push so every commit still gets a status without assembling anything.

repository_dispatch (type component-released) is kept as an optional, opt-in trigger for a component's release or tag boundary only; nothing sends it today.

The components are public, so the git dependencies uv resolves and the pack clones the runner makes need no credentials and the workflow needs no secret.

Licence

Copyright (c) 2026 Morten Olav Hansen. All rights reserved. See LICENSE.

The source is published for reference only: no licence to use, copy, modify or distribute it is granted, and any use beyond reading requires written permission.