dst-parquet

Plans and builds a long-term Parquet layout for a raw DST register delivery:

<delivery>/<provider>/<register>/year=<YYYY>/<source file>.parquet

for example 2025/dst/bef/year=2015/bef201512.parquet. A register whose files carry no year keeps everything except the year= level.

The delivery it was written against is 3,032 GB in 1,469 files. Converting that is a day's work nobody wants to do twice, and the decisions that would make it wrong are knowable in advance from the filenames and the headers. So there are two halves over one set of rules. plan says where every file would go and what it could not decide alone. convert does the work, and refuses any register the plan left blocked.

It is a single binary. It was first written in Python, and this is that program ported to Rust: the same commands, flags, rules file and layout, with every plan file the same byte for byte. The conversion reads each source once, puts it into its register directory's schema, writes it and measures it in the same pass, and holds about a gigabyte however large the source is.

Install

pip install dst-parquet-layout

The wheel holds the binary and nothing else, so it needs no other package. The name is the one the Python implementation was published under, so a line that installed that installs this, [convert] included.

dst-parquet-system-heap is the same program with the platform's own memory allocator in place of mimalloc. It exists to time the two against each other on the machine that runs the conversion.

Use

Review the rules against every naming shape in a delivery, from an exported register map, with no server access:

dst-parquet plan --map .../2026_register_map --out-dir plan/

Plan every file, from a per-file list or from a walk of the delivery. A walk reads each file's header for its row count:

dst-parquet plan --files files.csv --out-dir plan/
dst-parquet plan --root E:/rawdata/<delivery> --out-dir plan/

Date a delivery directory whose name does not say when it arrived:

dst-parquet dates --root E:/rawdata/<delivery> --out-dir dates/
dst-parquet dates --dates dates/DELIVERY_DATES.csv

Convert, once the plan is clean:

dst-parquet convert --root E:/rawdata/<delivery> --out E:/workdata/<project>/parquet \
    --plan plan/ --jobs 4

On a terminal, convert keeps a panel at the bottom of the window: the share of bytes done, the rate over the last minute, the time left, and a line per file being read. Each register directory leaves a line above it when it finishes. With its output sent to a file it prints those lines as they happen and the panel's figures every five minutes. --quiet prints only the summary.

Name one copy of each snapshot across deliveries, and compare the others:

dst-parquet resolve --out E:/workdata/<project>/parquet --out-dir resolution/

plan writes six files. TREE.txt is the whole layout as a directory tree, one line per register directory, which is the form to hand somebody deciding whether this is the structure they want. SUMMARY.txt is the status of every register directory and every open decision. REGISTER_PLAN.csv is one row per destination directory, PLACEMENT.csv one row per source file, and REVIEW.csv the decisions that need a yes and the ones that block. PROVENANCE.json records the version, the time, and digests of the rules and the file list.

Both commands exit non-zero when something is wrong: plan when a register is blocked, so it can gate a conversion in a script, and convert when it skipped a register or found something wrong in what it wrote.

What a conversion is allowed to do

convert takes --plan, and refuses to run if the rules or the file list have changed since that plan was written. A plan is read by people and acted on by a program, sometimes weeks apart, and nothing about editing rules.toml in between leaves a mark on the plan. --accept-rules-change overrides it, out loud.

Every source is read once, and what is checked is checked as the rows go past:

• The rows written are compared with the count in the source's header, less any rows the header marks deleted. A header that declares exactly 4,294,967,295 rows has a saturated counter, which says nothing about the file: it is read to its last page and reported, not compared.
• Every string cell is checked for U+FFFD, the replacement character a lossy decode leaves behind, which no later repair can undo. The reader decodes every windows-1252 byte, so in this delivery it can only appear in a file that is not what it declares. A file holding one stays written and fails the run. --verify-utf8 is still accepted, and no longer needed.
• A stock series' key, PNR for bef, is checked for a person held twice.
• For a cumulative series, the file kept for a year is asked how many of its rows fall before each dropped file's cut-off, against the count in the dropped file's header.
• Every file gets a fingerprint of its content, independent of row order and taken column by column, which resolve compares across deliveries.

What was measured goes into the register directory's _layout.json, and into each file's own Parquet footer, so a file whose sidecar is lost still says how many rows it holds. A file that is already there is not read again, so a run that resumes it reports again whatever the run that wrote it found wrong, marked as an earlier run's finding, and fails as that run did.

A run is resumable: a file that is already there is left alone, and the partial files an interrupted run left behind in the directories being converted are swept first. --jobs converts several register directories at once, largest first. --threads caps what the whole run uses, decoding and encoding together.

A register directory is written once and then left alone. Each source's size and the creation stamp in its SAS header are recorded beside the data, and a conversion whose sources no longer match refuses and names the files that differ. --new-version bef then writes the corrected data to the next free bef_v2 beside the directory already there, which is left exactly as it is.

--io-backend chooses how sources are read. buffered, the default, reads each file in large pieces on several threads at once. mmap maps the file into memory instead, and auto maps a local file and reads a network share in pieces. A conversion reads every byte of a file once, front to back, which is what large reads in parallel suit.

How it is written

Files are written with zstd at the level the rules set. Floats are split by byte stream and dates and times delta-encoded, and a column nearly every value of which is distinct in its first rows does without a dictionary, which is what lets those encodings apply. On a synthetic bef-shaped file that made the output 28 MB where the Python implementation's polars writer made 72 MB.

Column types are the reader's own, carried through, with one exception. A SAS TIME column is seconds since midnight, and the reader gives it as a duration, because nothing stops a SAS time from being negative or a day and more. It is written as a time of day, which R, polars and DuckDB read as a clock time; a value outside 00:00 to 24:00 is written as null, and the summary and the register's _layout.json say how many, in which file and column. The rule [layout] sas_time = "duration" writes every value as it is instead. Where the rules promote a code column from Float64 to String, a whole number is written without its decimal point: 1301.0 is "1301".

A tree converted before SAS times became times of day holds them as durations.

dst-parquet retime --out E:/workdata/<project>/parquet

rewrites each file that has such a column, in place, with the same rule for values outside a day, and records the counts in the register's _layout.json. A time hashes as the duration it was, so a file's fingerprint only changes where a value became null. --dry-run reports what it would change and writes nothing.

What is still open

OPEN_ITEMS.md, beside this file, is the list: the questions only a person can answer, the decisions the rules make that want a yes, and what is left undone in the code on purpose.

The rules

rules.toml holds every judgement, one entry each: which directory is which delivery and provider, which files are the same table, which registers are not partitioned by year, how a type conflict is promoted, which collisions are resolved and how, and what several files in one year of a register are to each other. The binary carries the file it was built with, and --rules PATH reads another. Changing a decision is a change to that file, not to the program, and tests/ pins the current answers, so an edit says out loud which files it moved.

Changing a few of them

A delivery that does not fit the rules is blocked in the plan, and the reason says which rule to change and what to write. To see what decides a register or a file, and the lines that would change it:

dst-parquet rules explain forloeb --plan plan/
dst-parquet rules explain bef201912.sas7bdat

The changes go in an overrides file, which holds only the entries it changes, in the sections rules.toml uses:

[undated.partition_on]
forloeb = "DATO_HENVISNING"

[layout]
compression_level = 6

A key replaces the key it names. An entry in a list, such as a [[series]] or a [[delivery]], is matched by what identifies it (register, snapshot) and only the fields given change; remove = true deletes it, and a new [[register]] is tried before the built-in ones. A register named under one of [undated]'s treatments leaves the others. Check the file, and see what it changes, before planning with it:

dst-parquet --overrides overrides.toml rules check
dst-parquet --overrides overrides.toml plan --root E:/rawdata/<delivery> --out-dir plan/
dst-parquet --overrides overrides.toml convert --root ... --plan plan/

The plan lists the changes in SUMMARY.txt, keeps a copy as OVERRIDES.toml and pins the rules and the overrides together, so a conversion without the same overrides is refused and told which flag it is missing. dst-parquet rules show prints the built-in rules, comments and all, which is where every default and its reason is written down.

What the rules have to survive

Each of these is observed in an August 2026 walk of one delivery.

The register is not the alphabetic prefix of the filename. soma_l2_t19_proc19 and soma_l3_t19_diag19 share a prefix and are different tables. Read as one register they give a schema in which 197 of 197 columns go missing somewhere.

115 files carry no year, for three different reasons: code lists with no time dimension, event tables pooled across the whole period, and tables the provider split into numbered shards. Each needs a different answer, and a register directory that is partitioned for some of its files and not others cannot be scanned at all.

A pooled table carries no year in its filename and thirty of them in its rows. [undated.partition_on] names the column for 27 of them, and they are cut into the same year= shape as every dated register. Where a register offers both dato_x and datotidspunkt_x the date is chosen, because the datetime variants are absent from one of the three files. A row whose date is missing keeps a null year in the standard __HIVE_DEFAULT_PARTITION__ directory: it is never dropped and never given a year it does not have, and the count is reported.

dnt_1159.sas7bdat parses as the year 1159 under any rule that takes four digits and stops. soma_l2_t19_drgkont19 carries its year as 19. dimcancergruppering_icd10 carries no year at all. bef199312 carries a month that must not be lost.

350 register-years appear in more than one delivery and 139 of those disagree on row count, usually by a handful of rows. Deliveries stay apart and nothing is deduplicated across them. Two files that would land in one partition with nothing to tell them apart block until a rule says which wins.

A month in a filename is a period in one register and a cut-off in another, and the filename does not say which. bef200803 is the population at the end of March 2008 and all four of that year's files are wanted, where lmdb202406 is every prescription of 2024 up to the end of June and is already inside lmdb202412. So a register that arrives more than once a year declares its grain in [[series]], the plan checks that against the file sizes before acting on it, and a year holding more than one file whose register declares nothing refuses to convert.

A year in a filename is a reading of a filename, not a measurement. A file more than eight times the median file of its own register is reported with the ratio, because udd_grundskole_fravaer_kmd2024 is 39.7 million rows where that register's years run 3 to 5 million.

624 of 2,491 register-columns are absent from at least one file of their own register, and 721 columns move position between years. Schemas are therefore built by name, never by order, names are uppercased, and each register directory carries a _layout.json naming the files each column was absent from.

Every file declares windows-1252 and Parquet declares UTF-8. Conversion checks the declaration per file and refuses a register that declares anything else.

Development

cargo nextest run -p dst-parquet
cargo clippy -p dst-parquet --all-targets -- -D warnings
just dst-parquet-wheel      # the Windows and macOS wheels, sanitised and scanned
just dst-parquet-publish    # uploads them, after asking

The conversion tests write real .sas7bdat files with sas7bdat-writer and read them through the published reader, so they test the same decode a delivery goes through. tests/real_sources.rs converts a small synthetic delivery of six such files, written by tests/synthetic/ into each test's own directory, since no .sas7bdat file is committed.