ad-data-pf

Tables (and later figures) for a research project on job-ad amenities. The functions are written and tested locally against simulated data, then pip installed on and run on the project data..

All numbers in this package come from simulated data.

Use

from ad_data_pf import validation as val, save_tex, save_fig, setup_log

setup_log(log_folder, 'descriptives')      # logs the package version first
frames = {'All AKU\nrespondents': aku, 'All\njob ads': jobads,
          'Linked\njob ads': jobads_linked, 'Job ads linked\nwith AKU response': aku_linked}
save_tex(val.desc_table(frames), out / 'tab_base_rates.tex')
save_tex(val.omission_table(aku_linked, ci='wilson'), out / 'tab_validation_measures.tex')
save_fig(val.prevalence_figure(aku_linked, aku), out / 'fig_sensitivity_occ_prevalence.pdf')

Tables return a bare tabular; figures return a matplotlib Figure. Defaults (variables, labels, panel titles) sit at the top of each module and in ad_data_pf.labels, and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

Arguments

Each function fixes the layout of its exhibit: the panels, and the statistic each one computes. The arguments say only which columns go in and what they are called, in three plain forms:

• {column: label}, one row per column: t_vars={'any_T': 'Any non-standard hours', 'work_night': 'Night work'}.
• {column: splits}, one row per group of a column:
  • {code: label} matches the leading characters of a string column ({'1': 'Managers', '2': 'Managers'} on the 6-digit disco) and the value of any other ({True: 'Public', False: 'Private'}). Codes may share a label.
  • [(upper bound, label), ...] bins a numeric column, left-closed, None for no bound (SIZE).
• {label: frame} where the columns, rows or points are samples (desc_table, robustness_table, ladder_figure). The label is the column header, row label or axis label, and \n breaks it into lines. Make each frame with polars before the call: al.filter(c.months_since_hire <= 3), al.with_columns(any_T=c.any_T_sometimes).

Panel titles can be renamed and {} drops a panel. There are no package-specific spec objects and no polars expressions as arguments.

validation makes these exhibits:

Inputs are checked. The following raise an error:

• T or M that is not Boolean, or has nulls;
• a row whose columns exist in no frame;
• a split that no row falls in, or a group with no pairs;
• T and M labels that do not pair up.

Cells resting on fewer than 5 observations are left empty (DST disclosure). In splits, a null or NaN counts as missing: in polars NaN compares above every number, so it would otherwise land in the top bin.

CHANGELOG.md lists what to retype when moving to a new version.

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

ad_data_pf.run_tests({'aku': aku, 'jobads': jobads,
                      'jobads_linked': jobads_linked, 'aku_linked': aku_linked})

It prints pytest's output, logs the summary line and returns 0 if every test passes. The frames are those after prep with all links kept, named as sim.simulate() names them. They are written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only, and '-x' stops at the first failure.

The data tests (tests/test_validation_data.py, marked data) run on these frames, and on simulated ones when there are none, as in CI. They check what the functions assume of their input but cannot check themselves, and build every exhibit with its defaults:

• T and M flags are Boolean without nulls, and any_T and any_M are the any of the flags in labels.py;
• disco and disco_jobad are 6-digit strings;
• public is Boolean and empl_sum a number of at least 0;
• every pair has a cvrnr and a prodnr, the bootstrap's clusters;
• jobads_linked is part of jobads, and aku_linked of aku and of jobads_linked;
• every default exhibit builds, its counts add up, and its intervals contain its estimates.

The other tests check the code on fake data. They give the same result anywhere, so on the server they test the environment.

A failed data test shows the offending values: the most common bad codes, or the first rows without a match.

Layout

src/ad_data_pf/
├── functions/          one module per topic: validation.py, ...
│                       imported from the top: from ad_data_pf import validation
├── simulation/         fake data per module (validation.py) and the files in data/
├── examples/
│   └── validation/     code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import ad_data_pf, pathlib; print(pathlib.Path(ad_data_pf.__file__).parent)"
python <that folder>/examples/validation/code/run_validation.py <output folder>

Every example also runs in the VS Code Interactive window (or any Jupyter kernel), whole or cell by cell (# %%). It writes to its own logs/ and output/, or to ./ad_data_pf_examples/<name>/ in the working directory if the installed package is read-only.

Adding a module

1. functions/<name>.py: importable at once as ad_data_pf.<name>; arguments in the forms of Arguments
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), never from sys.argv or __file__, and split sections with # %%. tests/test_examples.py runs every example as a script and as in a Jupyter kernel.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames but cannot check themselves (see Tests on the server data). Its frames fixture takes data_folder: sim.load(data_folder), or sim.simulate() when it is None.

Development

The package lives in ad_data_pf/ of the (private) project repository; run everything below from that folder.

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/ad_data_pf/simulation/validation.py   # save the fake data to simulation/data
uv run pytest --data src/ad_data_pf/simulation/data     # data tests on saved frames

The lower bounds in pyproject.toml are older versions. To test against exactly those, use a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push that touches ad_data_pf/ (.github/workflows/ad_data_pf_test.yml at the repository root).

Release

Bump version in pyproject.toml and add an entry to CHANGELOG.md. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag ad_data_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/ad_data_pf_publish.yml):

git tag ad_data_pf-v0.1.1 && git push origin ad_data_pf-v0.1.1