dst-parquet-layout

Plans and builds a long-term Parquet layout for a raw DST register delivery:

<delivery>/<provider>/<register>/year=<YYYY>/<source file>.parquet

for example 2025-05/dst/bef/year=2015/bef201512.parquet. A register whose files carry no year keeps everything except the year= level.

The delivery it was written against is 3,032 GB in 1,469 files. Converting that is a day's work nobody wants to do twice, and the decisions that would make it wrong are knowable in advance from the filenames and the headers. So the package has two halves over one set of rules. plan says where every file would go and what it could not decide alone. convert does the work, and refuses any register the plan left blocked.

Install

pip install dst-parquet-layout            # planning only, no dependencies
pip install "dst-parquet-layout[convert]" # adds polars and the SAS reader

The planning half is standard library only, on purpose. The people who have to agree with a layout should be able to read the plan without a numeric stack, a compiled reader, or a working package mirror behind a research firewall.

Use

Review the rules against every naming shape in a delivery, from an exported register map, with no server access:

dst-parquet plan --map .../2026_register_map --out-dir plan/

Plan every file, from the map's per-file detail table or from a live walk:

dst-parquet plan --files detail/files.csv --out-dir plan/
dst-parquet plan --root /data/rawdata/<delivery> --out-dir plan/

Date a delivery directory whose name does not say when it arrived:

dst-parquet dates --root /data/rawdata/<delivery> --out-dir dates/
dst-parquet dates --dates dates/DELIVERY_DATES.csv

Every .sas7bdat carries the timestamp of the SAS session that wrote it, which is when the extract was run rather than when it was copied. An extraction takes days and deliveries are months apart, so sorting the timestamps and cutting at the large gaps recovers the deliveries. It proposes a [[delivery]] entry only where the clusters reproduce the months the dated directories are already named after.

Convert, once the plan is clean:

dst-parquet convert --root /data/rawdata/<delivery> --out /data/parquet \
    --plan plan/ --files detail/files.csv --jobs 4

plan writes six files. TREE.txt is the whole layout as a directory tree, one line per register directory, which is the form to hand somebody deciding whether this is the structure they want. SUMMARY.txt is the status of every register directory and every open decision. REGISTER_PLAN.csv is one row per destination directory. PLACEMENT.csv is one row per source file. REVIEW.csv is the decisions that need a yes and the ones that block. PROVENANCE.json records the tool version, the time, and digests of the rules and the file list.

Both commands exit non-zero when something is wrong: plan when a register is blocked, so it can gate a conversion in a script, and convert when it skipped a register or wrote a file whose row count disagrees with its header.

What a conversion is allowed to do

convert takes --plan, and refuses to run if the rules or the file list have changed since that plan was written. A plan is read by people and acted on by a program, sometimes weeks apart, and nothing about editing rules.toml in between leaves a mark on the plan. Without the check the conversion quietly builds a tree nobody agreed to. --accept-rules-change overrides it, out loud.

Every written file is checked against the row count its source header declared, and the count that was actually written is recorded in the register directory's _layout.json beside the rules digest that produced it. --verify-utf8 reads each output back and counts U+FFFD, the replacement character a lossy decode leaves behind, which no later repair can undo.

A run is resumable: a file that is already there is left alone, and the partial files an interrupted run left behind are swept before anything new is written. --jobs converts several register directories at once; they share no state and no output path.

What is still open

OPEN_ITEMS.md, beside this file in the repository, is the list: three questions only a person can answer, seven decisions the rules already make that want a yes, nine things left undone in the code on purpose or since closed, and the fact that none of it has run against a real delivery yet.

The rules

src/dst_parquet_layout/rules.toml holds every judgement, one entry each: which directory is which delivery and provider, which files are the same table, which registers are not partitioned by year, how a type conflict is promoted, and which collisions are resolved and how. Changing a decision is a change to that file, not to the program. tests/ pins the current answers, so an edit says out loud which files it moved.

What the rules have to survive

Each of these is observed in an August 2026 walk of one delivery.

The register is not the alphabetic prefix of the filename. soma_l2_t19_proc19 and soma_l3_t19_diag19 share a prefix and are different tables. Read as one register they give a schema in which 197 of 197 columns go missing somewhere.

115 files carry no year, for three different reasons: code lists with no time dimension, event tables pooled across the whole period, and tables the provider split into numbered shards. Each needs a different answer, and a register directory that is partitioned for some of its files and not others cannot be scanned at all.

A pooled table carries no year in its filename and thirty of them in its rows. [undated.partition_on] names the column for 27 of them, taken from the register map and each present in every file of its register, and they are cut into the same year= shape as every dated register, in the streaming engine, without the whole table being held. Where the register offers both dato_x and datotidspunkt_x the date is chosen, because the datetime variants are absent from one of the three files. The choice is the event and not the report: a reporting timestamp would put a 2019 birth in year=2021 because that is when somebody typed it in.

The rest are written whole and [undated.written_whole] records why, so a table with no date column at all reads as settled rather than as an outstanding decision. A row whose date is missing keeps a null year in the standard __HIVE_DEFAULT_PARTITION__ directory: it is never dropped and never given a year it does not have, and the count is reported.

dnt_1159.sas7bdat parses as the year 1159 under any rule that takes four digits and stops. soma_l2_t19_drgkont19 carries its year as 19. dimcancergruppering_icd10 carries no year at all. bef199312 carries a month that must not be lost.

350 register-years appear in more than one delivery and 139 of those disagree on row count. lmdb 2023 is 29.2M rows in POPC and 58.2M in POPFULD, because POPFULD is the unfiltered extract and POPC is the cohort. Deliveries stay apart and nothing is deduplicated. Two files that would land in one partition with nothing to tell them apart block until a rule says which wins.

624 of 2,491 register-columns are absent from at least one file of their own register, and 60 header changes fall across 29 registers. Under one schema each absence becomes a null, which is a different claim, so each register directory carries a _layout.json naming the files each column was absent from.

722 column names are not all-uppercase and 37 appear in both cases inside one register, so lmdb offers CPRTJEK and cprtjek as two variables. 721 columns move position between years. Schemas are therefore built by name, never by order, and names are uppercased. Five columns change dtype across their years and are promoted, Float64 and String to String because all four such columns hold codes.

Every file declares windows-1252 and Parquet declares UTF-8. Conversion checks the declaration per file and refuses a register that declares anything else.

Getting it onto the analysis server

Either route works. The wheel installs from a package mirror, and the package has no dependencies unless the convert extra is asked for. Failing that, the whole package fits through tools/typed_transfer, which compresses a package into one checksummed script to type in.

Development

pytest                 # 88 tests
ruff check src tests
mypy

Tests run against the source tree without an install, and against the rules that ship with the package rather than a fixture copy.

Two of them cover the conversion. test_convert.py uses a stand-in for the SAS reader, which needs nothing installed and proves the code agrees with itself. test_real_sources.py runs against six real .sas7bdat files under tests/fixtures/, read by sas7bdat_polars, which is the only way to find out whether it agrees with a .sas7bdat: that windows-1252 comes through as UTF-8 with æøå intact, and that a SAS date, which is a float counting days from 1960, arrives as a date that can be partitioned on. It skips where the reader is not installed.

The fixtures are committed so the tests need no Rust toolchain. Regenerate them with the writer that made them, from the repository root:

cargo run -p sas7bdat-writer --example pooled_fixture -- \
    tools/dst-parquet-layout/tests/fixtures

The writer stamps a fixed creation time, so a re-run is an empty diff.