occu-choice-pf

Tables for the survey experiment of Perceived Wage Differences and Early Career Job Search Choices, run on the Statistics Denmark (DST) microdata. The functions are written and tested locally against simulated data, then pip installed on the server, where only loading, cleaning and the calls are typed by hand.

All numbers in this package come from simulated data.

Use

from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS

Tables return a bare tabular in the style of the paper's tables (the \sym stars of esttab included); save_tex writes it and logs it, so it can be read before an export request. Defaults (variables, labels) sit in occu_choice_pf.labels and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

The table of controls

reg_control_table(long) regresses each outcome on T_vj_vij_job, one panel per outcome, with one table column per entry of cols_control. Every column has coefs (reported), common_control, and the controls of its entry. The defaults (labels.COLS_CONTROL, labels.COMMON_CONTROL) give:

A control is {check-mark row label: terms}, and a term is a column ('age'), a column made into a dummy for each value ('i.age', as Stata's i.), or a polars expression (c.vj_vij_job ** 2). Your own columns:

from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])

The arguments, the check-mark rows and the input checks are in help(reg.reg_control_table); the estimation, and what drops out of a column (all of it logged), in help(reg). reg.reg_control_fits(long, ...) returns the pyfixest fits behind a table, {outcome: [fit per column]}, to look into with .summary() or pf.etable. examples/ has custom calls: a subsample, other regressors, clustering and a second fixed effect.

The long frame

One row per person and job type (pnr x type_sizerank_job), with the columns of misc_docs/all_waves_long_variable_definitions.txt: T, vj_vij_job, T_vj_vij_job, the pre-treatment variables in labels.PRETREATMENT, educ_field, disced_6d, and the outcomes in labels.OUTCOMES: spell_1_new_job (1 for the job type of the first new job after the survey) and appl_job (the number of applications to the job type).

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

occu_choice_pf.run_tests({'long': long})

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The frame is written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs.

The data tests (tests/test_regressions_data.py, marked data) check what the table assumes of its input but cannot check itself, and build both default tables:

• one row per person and job type;
• the outcomes, coefficients and controls are numbers or Booleans;
• T, the pre-treatment controls and the education columns take one value per person;
• T is 0 or 1, and T_vj_vij_job is T * vj_vij_job, missing (null or NaN) where vj_vij_job is;
• the pre-treatment controls have no missing values, so that the columns of a panel rest on the same rows but for what the Poisson regression drops;
• the outcomes are at least 0;
• both default tables build and keep T_vj_vij_job and vj_vij_job in every column, and pyfixest's estimation sample, whose persons the Individuals row and min_n count, has the fit's N rows (a check on the server's pyfixest version).

A failed data test shows the offending values. The other tests check the code on fake data; they give the same result anywhere, so on the server they test the environment.

Layout

src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       imported from the top: from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>

Every example also runs in the VS Code Interactive window, whole or cell by cell (# %%).

Adding a module

1. functions/<name>.py: importable at once as occu_choice_pf.<name>; arguments as {column: label} dicts, with defaults in labels.py
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), and split sections with # %%. tests/test_examples.py runs every example.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames.

Development

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames

The lower bounds in pyproject.toml are the server's versions for loguru, numpy and polars (as for ad-data-pf). To test against exactly those, in a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push (.github/workflows/test.yml).

Release

Bump version in pyproject.toml, run uv lock, and turn the ## Unreleased section of CHANGELOG.md into the version's entry. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag occu_choice_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/publish.yml):

git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0

The same tag runs .github/workflows/publish_1.yml, which publishes the same code as occu_choice_pf_1. The DST server caches a package when it is installed and won't update that name for 24 hours, so install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z

Both install the import occu_choice_pf, and the most recent install is the one used.

Before the first release, register both names as pending trusted publishers on pypi.org (Account settings, Publishing): owner AskerNC, repository occu_choice_pf, environment pypi, and workflow publish.yml for occu-choice-pf, publish_1.yml for occu_choice_pf_1.