dansk-register

A platform for epidemiological studies on the Danish national registers, and the study pipelines built on it.

The package contains no data. It is the code that reads a register delivery inside a secure research environment, builds cohorts and outcomes from it, and estimates and reports the results.

What is in it

The dependency direction is one way and enforced by a test: the platform never imports a study, and studies never import each other. Code that turns out to be general is lifted into dansk_register rather than shared sideways.

Layout

This directory is the private half of the PhD: the register platform, the three study pipelines, and the crates they call. Public methods crates are dependencies, not subdirectories. See ../admin/plans/code_consolidation.md for why, and for the migration that produced this shape on 26 August 2026.

• src/ - the Python packages above plus parental_income (Study I) and register_studies (the orchestration CLI), and src/rust/ for the pyo3 extension. All of it ships in one maturin wheel; the platform is not separately installable.
• crates/ - the Rust the studies call. register_runner (Parquet in, estimators out), cost_engine, cpi_deflate and person_years (the three bridge CLIs), sas7bdat-writer (the only thing that writes the format the Danish register hosts deliver), register-delivery (the synthetic delivery the end-to-end tests run on), and sporskifte (track changes for Typst).
• checkouts/ - gitignored reading copies of the public repos, cloned by just checkouts. Never a build input.
• docs/ - implementation documents only. Science goes to ../studies/.

Six of the seven crates are members of one cargo workspace with one Cargo.lock. sporskifte is not: it is a workspace of its own, and flattening it would put a vello/wgpu graphics stack and three vendored pdf-rs forks into this lockfile, which every --workspace command in the gate would then compile.

Dependencies that live elsewhere

The rule that produced this table: public code is consumed by version. If a fix is needed in one of these, fix it upstream, release it, bump the pin. Never vendor it, never path-depend across the boundary. cohort_matching was vendored for a week in August 2026 and the copy cost more than it bought.

did_methods stays a git pin because its honest feature builds HiGHS from C++, which is more than a crates.io release is currently worth. scd could follow the other three to a version pin; 0.1.3 is published.

Install

pip install dansk-register-platform                 # everything, all three studies
pip install "dansk-register-platform[study2]"       # names a study; installs the same files
pip install "dansk-register-platform[all]"          # all three, explicitly

The distribution was renamed from dansk-register at 0.7.0, to get past the analysis server mirror's 24-hour cache. The import packages are deliberately unchanged, so nothing in this codebase or on the server imports anything different; only pip install names the new one. The old name still resolves, to a distribution that stopped being updated before the rename.

The estimators are written in polars and validated against R, so the pipeline carries no numeric stack of its own. matplotlib is a base dependency because numpy arrived on every real run anyway; what keeps the boundary real is that its imports live under a figures package and a contract test fails the build if one escapes into an estimator. Without a renderer every result is still produced and the figure stage records that it could not draw.

repair is the only optional dependency: pyarrow, for register-repair-utf8 alone. Install it with pip install 'dansk-register-platform[repair]'. The same kind of contract test keeps pyarrow out of everything else.

Commands

edu-labour --list-profiles                          # what each profile runs
edu-labour-preflight --raw-root PATH                # what a delivery contains
edu-labour --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --resume
edu-labour-figures --bundle RUN/outputs/export --out figures

societal-costs --capability PATH                     # what this study needs vs. what a delivery has
societal-costs --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --diagnostics
societal-costs secondary_sensitivity --output-dir RUN   # the arm no profile runs

register-repair-utf8 PATH --report utf8_scan.csv    # find invalid UTF-8 in converted Parquet
register-repair-utf8 PATH --fix --report utf8_fix.csv

--allow-expensive-stages is required for production: matching scans the whole delivery, and without the flag the runner refuses rather than silently starting a multi-hour match. --profile production runs 19 stages but not secondary_sensitivity, the analyse-only sensitivity and subgroup arm, which reads a completed run and writes into a subdirectory of it.

register-repair-utf8 needs the repair extra and says so if it is missing.

Add --dry-run to any edu-labour invocation to resolve the plan and settings without touching data.

Design notes

Stages talk through files. Every stage writes named artifacts into a run directory and reads its inputs from there, so any stage can be re-run alone against a previous run's outputs, and --resume can skip what is already done.

Estimators are written against polars. Aalen-Johansen, Fine-Gray, Cox and the multi-state occupancy are implemented directly rather than taken from a modelling library, and each is validated against reference values from R with those values checked into the tests. That keeps the numbers checkable line by line and avoids a dependency that may not install in an offline environment.

Disclosure control is part of the pipeline. Results leave the secure environment as a bundle of CSV tables with small cells suppressed and the estimates resting on them blanked. Figures are rendered from that bundle rather than from the raw artifacts, so a figure cannot show what the table beside it was not cleared to show.

What this covers, and what it does not

Secure-environment research is usually described with the Five Safes, and it is worth saying plainly which of the five a piece of software can touch at all.

"Partly" is the operative word in both rows. The output checks are cell-by-cell and cannot see a small number recovered by differencing across tables; each bundle's MANIFEST.txt says so in its own text. Running this framework is not a substitute for your host's output review, and it does not make an unapproved analysis approved.

Working on it

just lists everything. The gates, which are what CI runs:

just lint          # both languages: ruff + three type checkers, rustfmt +
                   # clippy, and the three supply-chain audits
just rust-test     # 207 Rust tests across the three crates, via nextest
just test          # 2,953 Python tests, parallel
just gate          # all three, in CI's order

and the loops:

just test-fast     # 2,703 of them in ~15 s; everything but the real pipelines
just test-e2e      # the 190 that run a real pipeline over a generated delivery
just types         # ty + pyrefly: 0.7 s from cold, against mypy's 6 s
just typos         # spell-check; not a gate, see _typos.toml
just mutants       # would the tests have noticed? Slow, not a gate
prek install       # optional: run the formatters before each commit

and the escape hatch, for when a fix has to be tried in a public crate before it is released:

just checkouts     # clone or fast-forward the public repos in checkouts.toml
just patch-on      # build against those checkouts, not the published versions
just patch-off     # back to the pins

patch-on spans both toolchains, because the coupling does: a cargo [patch] for cohort_matching, semicontinuous_models, scd and did_methods, and a [tool.uv.sources] editable path for sas7bdat-polars, which reaches the pipeline as a PyPI wheel rather than as a cargo dependency. It dirties Cargo.toml, pyproject.toml and both lockfiles, deliberately: a patched tree builds code that is in no lockfile, and that should be visible. just patch-off restores all four, and is what to run before committing, and before trusting a test result you intend to act on.

Two things worth knowing before the first full run:

The generated delivery is cached, not rebuilt. The end-to-end tier runs against an 854 MB synthetic delivery, and it is byte-reproducible from its inputs, so it is kept under target/test-deliveries/ keyed by a hash of the generator, both catalogs and the parameters. It is built once and shared by every xdist worker and every later run. REGISTER_DELIVERY_CACHE= (empty) turns that off. See tests/full_delivery.py for why it is a cache rather than the usual filelock recipe.

One pytest at a time. Overlapping full runs is what filled the disk on 25 August 2026. The suite now refuses to start the end-to-end tier below 8 GB free, and says how to reclaim; REGISTER_STUDIES_MIN_FREE_GB=0 overrides it.

Status

Research code, developed alongside the studies it implements. The public API is not stable.

Documentation

docs/README.md is the index, and it says which document is a standing contract and which is a finished pass kept because the code cites it. Two worth naming: docs/open_questions.md is the open-items register, and docs/REUSABILITY_ANALYSIS.md is why the platform is shaped the way it is.

Knowledge rather than implementation — register semantics, methodology, study protocols — lives in the PhD store, not here. docs/README.md states the rule that decides which is which.

Licence

MIT. See LICENSE.