Skip to main content

dst-parquet

Converts a raw delivery of SAS register files from Statistics Denmark into a tree of Parquet files, which R, Python and Stata can read. Each SAS file becomes one Parquet file, in a folder per register:

<delivery>/<provider>/<register>/year=<YYYY>/<source file>.parquet

for example 2025/dst/bef/year=2015/bef201512.parquet. Only a register whose SAS files are split by year gets the year= folders. A register delivered as one file is written as one file.

Install

pip install dst-parquet-layout

This installs the command dst-parquet, and nothing else is needed. dst-parquet --version checks that it is there.

Converting a delivery

Commands are shown on one line, which is how cmd and PowerShell take them. Put a path in quotes when it holds a space.

First a dry run, which reads the SAS headers and writes nothing:

dst-parquet convert --root "E:\rawdata\project" --out "E:\workdata\project\data" --dry-run

--root is the folder the delivery's .sas7bdat files are under, and --out is where the tree is written. Then the conversion itself:

dst-parquet convert --root "E:\rawdata\project" --out "E:\workdata\project\data"

A full delivery takes about a day. While it runs, a panel at the bottom of the window shows how far it has come and how long is left. The last line starts with RESULT and says whether everything was converted, and what to do if not.

Stopping is safe. Ctrl+C, a closed window or a restart loses only the files being written at that moment. Run the same command again and it continues: files already written are left as they are, and half-written ones are removed.

Other folders and files in --out are left alone. Files being written have names that start with _, which the Parquet readers of R and Python skip, so the tree can be read while a run is going. Only one run at a time can write to a tree, and a second one is refused.

What you get

Each register folder holds its Parquet files and a _layout.json. That file records which SAS file each Parquet file came from, with its full path, the row counts, anything found wrong, and the release and rules that wrote it.

Column names are in lowercase. Columns are matched across years by name, so a column that is missing from one year is there as empty values in that year's files, and _layout.json names the files it was missing from.

Columns whose names start with _ were added by the conversion and are not in the raw data:

  • _src_file, in every file: the name of the SAS file the row came from. The same name recurs across deliveries, and _layout.json gives its full path.
  • _population, in registers from the two Grunddata extracts: popc or popfuld. The extracts overlap, so combining them counts people twice.
  • _src_month, for a register delivered several times a year: the month in the file name.
  • _ref_date, for bef: the date the population count refers to.
  • One for each part of a file name that tells otherwise identical tables apart, such as _tabel in the DRG tables.

When it refuses or reports a problem

Each refusal names the register and says what to do. The common ones:

  • An earlier conversion is there. The register was written by an earlier release or under other rules, or by a run that stopped before it finished. Add --replace, which removes what that run wrote in that register's folder and converts the register again. Nothing else is removed.
  • Another run is writing to the tree. Wait for it to finish, or stop it.
  • The sources differ from the ones converted. A SAS file was replaced or added since the register was converted. --new-version <register> writes the register again as <register>_v2 beside the old one, which is kept.
  • A register is blocked. The rules cannot place one of its files. dst-parquet rules explain <register> says which rule decides it and what to write in an overrides file to change it.
  • Row counts, repeated keys or U+FFFD. The files are written, but something in them needs a look: fewer or more rows than the SAS header says, a person twice in one population snapshot, or text that could not be decoded. The run counts as failed until somebody has looked.

Words used here and in the output

  • Delivery: one batch of files from Statistics Denmark or the Danish Health Data Authority, named by the year it arrived, such as 2025.
  • Provider: who sent it, dst or sds.
  • Register: one table, such as bef or lmdb, however many files it arrives in.
  • Register directory: the folder a register's Parquet files are written to.
  • Sidecar: the _layout.json in each register directory.
  • Snapshot: one file of a register delivered several times a year, such as bef at the end of each quarter.
  • Reference and pooled: registers with no year in their file names. A reference table is a code list, and a pooled table holds events from many years in one file. Both are written whole.
  • Rules: the built-in rules.toml, which decides all of the above. dst-parquet rules show prints it.

Other commands

convert works out the layout itself as it starts. plan does the same without converting anything, and writes six files to read or to send to somebody deciding whether this is the structure they want:

dst-parquet plan --root "E:\rawdata\project" --out-dir plan

TREE.txt is the whole layout as a directory tree, one line per register directory. SUMMARY.txt is the status of every register directory and every open decision. REGISTER_PLAN.csv is one row per destination directory, PLACEMENT.csv one row per source file, and REVIEW.csv the decisions that need a yes and the ones that block. PROVENANCE.json records the version, the time, and digests of the rules and the file list. Given --plan plan, a conversion refuses to run if the rules or the file list have changed since.

plan can also read a per-file list, or an exported register map, which needs no access to the server:

dst-parquet plan --files files.csv --out-dir plan
dst-parquet plan --map 2026_register_map --out-dir plan

dates dates a delivery directory whose name does not say when it arrived:

dst-parquet dates --root "E:\rawdata\project" --out-dir dates

resolve names one copy of each snapshot delivered more than once, and compares the others with it:

dst-parquet resolve --out "E:\workdata\project\data" --out-dir resolution

retime is a one-off repair for a tree written before SAS times were written as times of day, described under "How it is written" below.

plan exits non-zero when a register is blocked, so it can gate a conversion in a script, and convert when it skipped a register or found something wrong in what it wrote. A path that does not exist, or a register name the delivery does not have, stops either command before it starts.

Reference

The rest of this file is for whoever maintains the rules or wants to know how a conversion decides what it does.

The delivery the tool was written against is 3,032 GB in 1,469 files. It is a single binary, first written in Python and ported to Rust with the same commands, rules file and layout. The conversion reads each source once, puts it into its register directory's schema, writes it and measures it in the same pass, and holds about a gigabyte however large the source is. dst-parquet-system-heap is the same program with the platform's own memory allocator in place of mimalloc, for timing the two on the machine that runs the conversion.

With its output sent to a file, convert prints a line per register directory as it finishes and the panel's figures every five minutes. --quiet prints only the summary.

What a conversion checks and refuses

convert plans for itself from the rules it runs with, and refuses every register directory that plan blocks. Given --plan, it also refuses to run if the rules or the file list have changed since that plan was written. A plan is read by people and acted on by a program, sometimes weeks apart, and nothing about editing rules.toml in between leaves a mark on the plan. --accept-rules-change overrides it, out loud. Either way every register's _layout.json records the release and the rules that wrote it.

Every source is read once, and what is checked is checked as the rows go past:

  • The rows written are compared with the count in the source's header, less any rows the header marks deleted. A header that declares exactly 4,294,967,295 rows has a saturated counter, which says nothing about the file: it is read to its last page and reported, not compared.
  • Every string cell is checked for U+FFFD, the replacement character a lossy decode leaves behind, which no later repair can undo. The reader decodes every windows-1252 byte, so in this delivery it can only appear in a file that is not what it declares. A file holding one stays written and fails the run. --verify-utf8 is still accepted, and no longer needed.
  • A stock series' key, PNR for bef, is checked for a person held twice.
  • For a cumulative series, the file kept for a year is asked how many of its rows fall before each dropped file's cut-off, against the count in the dropped file's header.
  • Every file gets a fingerprint of its content, independent of row order and taken column by column, which resolve compares across deliveries.

What was measured goes into the register directory's _layout.json, and into each file's own Parquet footer beside the release and rules that wrote it. A file that is already there is not read again, so a run that resumes it reports again whatever the run that wrote it found wrong, marked as an earlier run's finding, and fails as that run did.

A run is resumable: a file that is already there is left alone, and the partial files an interrupted run left behind in the directories being converted are swept first. A file is written as _<name>.part, and a pooled source's year= files under _<stem>.part, and moved into place when it is finished. A register gets a _layout.json marked in_progress before its first file and the full one after its last, so a run cut off in between leaves a register the next run finishes, taking what the cut-off run found from the footers of the files it wrote. A run holds _dst-parquet.lock in --out while it writes, which the operating system releases when the run ends however it ends.

--jobs converts several register directories at once, largest first. --threads caps what the whole run uses, decoding and encoding together.

A register directory is written once and then left alone. Each source's size and the creation stamp in its SAS header are recorded beside the data, and a conversion whose sources no longer match refuses and names the files that differ. --new-version bef then writes the corrected data to the next free bef_v2 beside the directory already there, which is left exactly as it is.

A register directory written under other rules, as _layout.json records them, is refused too, and so is one holding Parquet files with no _layout.json that can be read, as a release before 0.2 leaves a register it was stopped in. --replace removes what that run wrote and converts the register again, so a release with new rules can be run over the tree an earlier one wrote:

dst-parquet convert --root "E:\rawdata\project" --out "E:\workdata\project\data" --replace --dry-run
dst-parquet convert --root "E:\rawdata\project" --out "E:\workdata\project\data" --replace

Only the Parquet files of the sources the register's _layout.json names, or this run places there, are removed, in that register's directory and its <register>_v<n> beside it, so nothing else under --out is touched. A Parquet file there that neither accounts for stops the register, with nothing removed, until it is moved away. The sidecar goes last, so a run cut off partway removes the rest next time. The rules include the overrides file, so changing either makes every register due for replacing.

--io-backend chooses how sources are read. buffered, the default, reads each file in large pieces on several threads at once. mmap maps the file into memory instead, and auto maps a local file and reads a network share in pieces. A conversion reads every byte of a file once, front to back, which is what large reads in parallel suit.

How it is written

Files are written with zstd at the level the rules set. Floats are split by byte stream and dates and times delta-encoded, and a column nearly every value of which is distinct in its first rows does without a dictionary, which is what lets those encodings apply. On a synthetic bef-shaped file that made the output 28 MB where the Python implementation's polars writer made 72 MB.

Column types are the reader's own, carried through, with one exception. A SAS TIME column is seconds since midnight, and the reader gives it as a duration, because nothing stops a SAS time from being negative or a day and more. It is written as a time of day, which R, polars and DuckDB read as a clock time; a value outside 00:00 to 24:00 is written as null, and the summary and the register's _layout.json say how many, in which file and column. The rule [layout] sas_time = "duration" writes every value as it is instead. Where the rules promote a code column from Float64 to String, a whole number is written without its decimal point: 1301.0 is "1301".

A tree converted before SAS times became times of day holds them as durations.

dst-parquet retime --out E:/workdata/<project>/parquet

rewrites each file that has such a column, in place, with the same rule for values outside a day, and records the counts in the register's _layout.json. --out may be the tree or any directory inside or above it: each register's files are found from where its own _layout.json sits, and one whose files are not there is named. Several files are rewritten at once (--jobs), each file's row groups are decoded in parallel, and the progress panel shows them as convert does. A time hashes as the duration it was, so a file's fingerprint only changes where a value became null. --dry-run reads only the duration columns, reports what it would change, and writes nothing.

What is still open

OPEN_ITEMS.md, beside this file, is the list: the questions only a person can answer, the decisions the rules make that want a yes, and what is left undone in the code on purpose.

The rules

rules.toml holds every judgement, one entry each: which directory is which delivery and provider, which files are the same table, which registers are not partitioned by year, how a type conflict is promoted, which collisions are resolved and how, and what several files in one year of a register are to each other. The binary carries the file it was built with, and --rules PATH reads another. Changing a decision is a change to that file, not to the program, and tests/ pins the current answers, so an edit says out loud which files it moved.

Changing a few of them

A delivery that does not fit the rules is blocked in the plan, and the reason says which rule to change and what to write. To see what decides a register or a file, and the lines that would change it:

dst-parquet rules explain forloeb --plan plan/
dst-parquet rules explain bef201912.sas7bdat

The changes go in an overrides file, which holds only the entries it changes, in the sections rules.toml uses:

[undated.partition_on]
forloeb = "dato_henvisning"

[layout]
compression_level = 6

A key replaces the key it names. An entry in a list, such as a [[series]] or a [[delivery]], is matched by what identifies it (register, snapshot) and only the fields given change; remove = true deletes it, and a new [[register]] is tried before the built-in ones. A register named under one of [undated]'s treatments leaves the others. Check the file, and see what it changes, before planning with it:

dst-parquet --overrides overrides.toml rules check
dst-parquet --overrides overrides.toml plan --root E:/rawdata/<delivery> --out-dir plan/
dst-parquet --overrides overrides.toml convert --root ... --plan plan/

The plan lists the changes in SUMMARY.txt, keeps a copy as OVERRIDES.toml and pins the rules and the overrides together, so a conversion without the same overrides is refused and told which flag it is missing. dst-parquet rules show prints the built-in rules, comments and all, which is where every default and its reason is written down.

What the rules have to survive

Each of these is observed in an August 2026 walk of one delivery.

The register is not the alphabetic prefix of the filename. soma_l2_t19_proc19 and soma_l3_t19_diag19 share a prefix and are different tables. Read as one register they give a schema in which 197 of 197 columns go missing somewhere.

115 files carry no year, for three different reasons: code lists with no time dimension, event tables pooled across the whole period, and tables the provider split into numbered shards. Each needs a different answer, and a register directory that is partitioned for some of its files and not others cannot be scanned at all.

A pooled table carries no year in its filename and thirty of them in its rows. It is written whole, because a register is partitioned by year only where its source is split into year tables. [undated.partition_on] can name a date column to cut one into the year= shape of a dated register. A row whose date is missing then keeps a null year in the standard __HIVE_DEFAULT_PARTITION__ directory: it is never dropped and never given a year it does not have, and the count is reported.

dnt_1159.sas7bdat parses as the year 1159 under any rule that takes four digits and stops. soma_l2_t19_drgkont19 carries its year as 19. dimcancergruppering_icd10 carries no year at all. bef199312 carries a month that must not be lost.

350 register-years appear in more than one delivery and 139 of those disagree on row count, usually by a handful of rows. Deliveries stay apart and nothing is deduplicated across them. Two files that would land in one partition with nothing to tell them apart block until a rule says which wins.

A month in a filename is a period in one register and a cut-off in another, and the filename does not say which. bef200803 is the population at the end of March 2008 and all four of that year's files are wanted, where lmdb202406 is every prescription of 2024 up to the end of June and is already inside lmdb202412. So a register that arrives more than once a year declares its grain in [[series]], the plan checks that against the file sizes before acting on it, and a year holding more than one file whose register declares nothing refuses to convert.

A year in a filename is a reading of a filename, not a measurement. A file more than eight times the median file of its own register is reported with the ratio, because udd_grundskole_fravaer_kmd2024 is 39.7 million rows where that register's years run 3 to 5 million.

624 of 2,491 register-columns are absent from at least one file of their own register, and 721 columns move position between years. Schemas are therefore built by name, never by order, names are lowercased, and each register directory carries a _layout.json naming the files each column was absent from.

Every file declares windows-1252 and Parquet declares UTF-8. Conversion checks the declaration per file and refuses a register that declares anything else.

Development

cargo nextest run -p dst-parquet
cargo clippy -p dst-parquet --all-targets -- -D warnings
just dst-parquet-wheel      # the Windows and macOS wheels, sanitised and scanned
just dst-parquet-publish    # uploads them, after asking

The conversion tests write real .sas7bdat files with sas7bdat-writer and read them through the published reader, so they test the same decode a delivery goes through. tests/real_sources.rs converts a small synthetic delivery of six such files, written by tests/synthetic/ into each test's own directory, since no .sas7bdat file is committed.

Metadata

Release files for dst-parquet-layout 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for dst-parquet-layout 0.2.0
File Interpreter ABI Platform Actions
dst_parquet_layout-0.2.0-py3-none-win_amd64.whl Python 3 none Windows x86-64 Details for dst_parquet_layout-0.2.0-py3-none-win_amd64.whl
dst_parquet_layout-0.2.0-py3-none-macosx_11_0_arm64.whl Python 3 none macOS 11.0+ ARM64 Details for dst_parquet_layout-0.2.0-py3-none-macosx_11_0_arm64.whl

Total release size: 13.8 MB

Release files / dst_parquet_layout-0.2.0-py3-none-win_amd64.whl

Download URL dst_parquet_layout-0.2.0-py3-none-win_amd64.whl
Size 7.3 MB
Tags Python 3 Windows x86-64
SHA-256 checksum
How to use checksums
f8890b2832e4219c4bfcbc29e7c589a24eac828968c2619c2b1471bf222ad3da
BLAKE2b-256 checksum
How to use checksums
da6b5240f71807e6dd31225da3352409e61879eb8a10ee0b073204234c49e339
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.10

Release files / dst_parquet_layout-0.2.0-py3-none-macosx_11_0_arm64.whl

Download URL dst_parquet_layout-0.2.0-py3-none-macosx_11_0_arm64.whl
Size 6.5 MB
Tags Python 3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
8104db2953349062e712a51154f41bca79c49080b20d867e4fb035257d5dd29e
BLAKE2b-256 checksum
How to use checksums
eac5f6ce925b7ee23cca0ec62fc62d4b4c119bc78e5feae51d19a0b7f903422d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.10

Security for dst-parquet-layout 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page