ad-data-pf

Tables (and later figures) for a research project on job-ad amenities. The functions are written and tested locally against simulated data, then pip installed on and run on the project data..

All numbers in this package come from simulated data.

Use

from ad_data_pf import validation as val, save_tex, save_fig, setup_log

setup_log(log_folder, 'descriptives')      # logs the package version first
frames = {'All AKU\nrespondents': aku, 'All\njob ads': jobads,
          'Linked\njob ads': jobads_linked, 'Job ads linked\nwith AKU response': aku_linked}
save_tex(val.desc_table(frames), out / 'tab_base_rates.tex')
save_tex(val.omission_table(aku_linked, ci='wilson'), out / 'tab_validation_measures.tex')
save_fig(val.prevalence_figure(aku_linked, aku), out / 'fig_sensitivity_occ_prevalence.png')

Tables return a bare tabular; figures return a matplotlib Figure. Figures are drawn in style/paper.mplstyle with the colours of style/ggplot2.mplstyle. save_fig(fig, path, presentation=True) draws a package figure again in style/presentation.mplstyle, for slides. Draw a figure of your own inside with figure_style(): (or figure_style(presentation=True)) to match. While a style is in use, matplotlib and fontTools log only warnings and errors. Defaults (variables, labels, panel titles) sit at the top of each module and in ad_data_pf.labels, and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

Arguments

Each function fixes the layout of its exhibit: the panels, and the statistic each one computes. The arguments say only which columns go in and what they are called, in three plain forms:

• {column: label}, one row per column: t_vars={'any_T': 'Any non-standard hours', 'work_night': 'Night work'}.
• {column: splits}, one row per group of a column:
  • {code: label} matches the leading characters of a string column ({'1': 'Managers', '2': 'Managers'} on the 6-digit disco) and the value of any other ({True: 'Public', False: 'Private'}). Codes may share a label.
  • [(upper bound, label), ...] bins a numeric column, left-closed, None for no bound (SIZE).
• {label: frame} where the columns, rows or points are samples (desc_table, robustness_table, ladder_figure). The label is the column header, row label or axis label, and \n breaks it into lines. Make each frame with polars before the call: al.filter(c.months_since_hire <= 3), al.with_columns(any_T=c.any_T_sometimes).

Panel titles can be renamed and {} drops a panel. There are no package-specific spec objects and no polars expressions as arguments.

validation makes these exhibits:

Exhibit	Function
Samples and base rates	desc_table (panels units, T, M, composition; any number of frames)
Confusion table	confusion_table (shares of N by default, or counts)
Validation measures by type	omission_table (Wilson or cluster-bootstrap intervals)
Measure by occupational prevalence	prevalence_figure (measure='sens' or 'fom')
Measures by group	heterogeneity_table (one panel per split column)
Measures by linkage criterion	ladder_figure
Robustness	robustness_table (one row per sample)

Inputs are checked. The following raise an error:

• T or M that is not Boolean, or has nulls;
• a row whose columns exist in no frame;
• a split that no row falls in, or a group with no pairs;
• T and M labels that do not pair up.

Cells resting on fewer than 5 observations are left empty (DST disclosure). In splits, a null or NaN counts as missing: in polars NaN compares above every number, so it would otherwise land in the top bin.

CHANGELOG.md lists what to retype when moving to a new version.

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

ad_data_pf.run_tests({'aku': aku, 'jobads': jobads,
                      'jobads_linked': jobads_linked, 'aku_linked': aku_linked})

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The summary at the end names each failed and skipped test, with the reason for a skip. The frames are those after prep with all links kept, named as sim.simulate() names them. They are written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only, and '-x' stops at the first failure. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs; that test runs locally.

The data tests (tests/test_validation_data.py, marked data) run on these frames, and on simulated ones when there are none, as in CI. They check what the functions assume of their input but cannot check themselves, and build every exhibit with its defaults:

• T and M flags are Boolean without nulls, and any_T and any_M are the any of the flags in labels.py;
• disco and disco_jobad are 6-digit strings;
• public is Boolean and empl_sum a number of at least 0;
• every pair has a cvrnr and a prodnr, the bootstrap's clusters;
• jobads_linked is part of jobads, and aku_linked of aku and of jobads_linked;
• every default exhibit builds, its counts add up, and its intervals contain its estimates.

The other tests check the code on fake data. They give the same result anywhere, so on the server they test the environment.

A failed data test shows the offending values: the most common bad codes, or the first rows without a match.

Layout

src/ad_data_pf/
├── functions/          one module per topic: validation.py, ...
│                       imported from the top: from ad_data_pf import validation
├── style/              matplotlib styles: paper, presentation, ggplot2 (colours)
├── simulation/         fake data per module (validation.py) and the files in data/
├── examples/
│   └── validation/     code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import ad_data_pf, pathlib; print(pathlib.Path(ad_data_pf.__file__).parent)"
python <that folder>/examples/validation/code/run_validation.py <output folder>

Every example also runs in the VS Code Interactive window (or any Jupyter kernel), whole or cell by cell (# %%). It writes to its own logs/ and output/, or to ./ad_data_pf_examples/<name>/ in the working directory if the installed package is read-only.

Adding a module

1. functions/<name>.py: importable at once as ad_data_pf.<name>; arguments in the forms of Arguments
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), never from sys.argv or __file__, and split sections with # %%. tests/test_examples.py runs every example as a script and as in a Jupyter kernel.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames but cannot check themselves (see Tests on the server data). Its frames fixture takes data_folder: sim.load(data_folder), or sim.simulate() when it is None.

Development

The package lives in ad_data_pf/ of the (private) project repository; run everything below from that folder.

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/ad_data_pf/simulation/validation.py   # save the fake data to simulation/data
uv run pytest --data src/ad_data_pf/simulation/data     # data tests on saved frames

The lower bounds in pyproject.toml are older versions. To test against exactly those, use a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push that touches ad_data_pf/ (.github/workflows/ad_data_pf_test.yml at the repository root).

Release

Bump version in pyproject.toml and add an entry to CHANGELOG.md. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag ad_data_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/ad_data_pf_publish.yml):

git tag ad_data_pf-v0.1.1 && git push origin ad_data_pf-v0.1.1

The same tag runs .github/workflows/ad_data_pf_publish_1.yml, which publishes the same code under a second name, ad_data_pf_1. The DST server caches a package when it is installed and won't update that name for 24 hours. On the server, install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install ad-data-pf==X.Y.Z
pip install ad_data_pf_1==X.Y.Z

Both install the import ad_data_pf, and the most recent install is the one used.