ad-data-pf

Tables (and later figures) for a research project on job-ad amenities, run on Statistics Denmark (DST) microdata. The functions are written and tested locally against simulated data, then pip installed on the DST research server, where the data are cleaned and the functions called.

All numbers in this package come from simulated data.

Use

from ad_data_pf import validation as val, save_tex, save_fig, setup_log

setup_log(log_folder, 'descriptives')      # logs the package version first
frames = {'aku': aku, 'jobads': jobads, 'jobads_linked': jobads_linked, 'aku_linked': aku_linked}
save_tex(val.desc_table(frames), out / 'tab_base_rates.tex')
save_tex(val.omission_table(aku_linked, ci='wilson'), out / 'tab_validation_measures.tex')
save_fig(val.prevalence_figure(aku_linked, aku), out / 'fig_sensitivity_occ_prevalence.pdf')

Tables return a bare tabular; figures return a matplotlib Figure. Defaults (variables, labels, panels) sit at the top of each module and in ad_data_pf.labels, and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

validation makes these exhibits:

Inputs are checked: T and M must be Boolean without nulls, a row whose columns exist in no frame or whose filter selects nothing raises, and cells resting on fewer than 5 observations are left empty (DST disclosure).

On the server, install an exact version (pip install ad-data-pf==0.1.0) so a rerun reproduces the same table.

Layout

src/ad_data_pf/
├── functions/          one module per topic: validation.py, ...
│                       imported from the top: from ad_data_pf import validation
├── simulation/         fake data per module (validation.py) and the files in data/
└── examples/
    └── validation/     code/ runs every table on the fake data;
                        logs/ and output/ hold what it writes

simulation/ and examples/ are installed with the package. To find them:

python -c "import ad_data_pf, pathlib; print(pathlib.Path(ad_data_pf.__file__).parent)"
python <that folder>/examples/validation/code/run_validation.py <output folder>

Without an output folder the example writes next to itself, which fails if the installed package is read-only.

Adding a module

1. functions/<name>.py: importable at once as ad_data_pf.<name>
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep
4. tests/test_<name>.py

Development

The package lives in ad_data_pf/ of the (private) project repository; run everything below from that folder.

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/ad_data_pf/simulation/validation.py   # redraw the shipped fake data

The lower bounds in pyproject.toml are the versions on the DST server. To test against exactly those, use a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push that touches ad_data_pf/ (.github/workflows/ad_data_pf_test.yml at the repository root).

Release

Bump version in pyproject.toml, commit, then push a tag ad_data_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/ad_data_pf_publish.yml):

git tag ad_data_pf-v0.1.1 && git push origin ad_data_pf-v0.1.1