occu-choice-pf

Tables for the survey experiment of Perceived Wage Differences and Early Career Job Search Choices, run on the Statistics Denmark (DST) microdata. The functions are written and tested locally against simulated data, then pip installed on the server, where only loading, cleaning and the calls are typed by hand.

All numbers in this package come from simulated data.

Use

from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.regression_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.regression_table(long, method='ols'), out / 'tab_ols.tex')     # OLS

Tables return a bare tabular in the style of the paper's tables (the \sym stars of esttab included); save_tex writes it and logs it, so it can be read before an export request. Defaults (variables, labels) sit in occu_choice_pf.labels and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

The regression table

regression_table(long) regresses each outcome on T_vj_vij_job and vj_vij_job with a person fixed effect, one panel per outcome, and adds controls column by column:

method='poisson' (the default) is pyfixest.fepois, PPML as ppmlhdfe in Stata; method='ols' is pyfixest.feols, as reghdfe. Standard errors are clustered on the person (cluster='pnr'), CRV1 with pyfixest's small-sample correction, and the stars (* p<0.1, ** p<0.05, *** p<0.01) use pyfixest's p-values. reg.estimate(long, ...) returns the pyfixest fits behind a table, {outcome: [fit per column]}, to look into with .summary() or pf.etable.

The arguments say which columns go in and what they are called:

• outcomes={column: panel title}: a count or a 0/1 dummy per person and job type; one panel each, lettered A), B), ... when there are several.
• coefs={column: row label}: regressors of every column, reported.
• base={column: row label}: regressors of every column, not reported but checked under Controls, as the paper does with the misperception. Below the panels, as in the paper, a Controls: row heads one check-mark row per control: base, the fixed effect, then each step.
• x='vj_vij_job': the column the powers and interactions are built from.
• powers={label: [2, 3]}, controls={label: [column, ...]}, dummies={label: column}: each entry adds one column, in that order, and its key labels the check-mark row. {} adds none. Powers and controls add up column by column; each set of dummies is an alternative, added on its own to the last of those columns. dummies makes a dummy for each value but the lowest.
• fe={column: label}: the one fixed effect; cluster: a column.
• \n in a label breaks it into lines.

What drops out of a column, all of it logged:

• rows with a missing (null or NaN) value in any variable of the column; a row with a null in a dummies column leaves from that table column on;
• singletons, and in the Poisson regression persons whose outcome is 0 in every row (they carry no information there; pyfixest logs them as removed "because of separation");
• in the Poisson regression, separated rows: an outcome of 0 that some combination of the regressors predicts exactly. They are found with an exact linear program (scipy) and dropped first, as ppmlhdfe does. The typical case is a person alone in a DISCED code whose outcome is at the extreme of vj_vij_job. Left in, they make pyfixest fail ("Demeaning failed");
• regressors collinear with earlier ones or with the fixed effect (some DISCED codes within a field). A reported coefficient that is dropped leaves its cells empty.

So the persons and rows below each panel can differ between panels and columns. Cells resting on fewer than min_n=5 persons are left empty (DST disclosure).

The functions take a polars DataFrame and work in polars; pyfixest gets a pandas copy of each regression's columns, the only use of pandas.

Inputs are checked. The following raise an error: a frame that is not a polars DataFrame (convert pandas with pl.from_pandas, collect a LazyFrame first), an outcome, coefficient or control that is not a number or Boolean (a categorical column goes in dummies), an infinite value, a dummies column with one value, more than one fixed effect, and an unknown method.

The long frame

One row per person and job type (pnr x type_sizerank_job), with the columns of misc_docs/all_waves_long_variable_definitions.txt: T, vj_vij_job, T_vj_vij_job, the pre-treatment variables in labels.PRETREATMENT, educ_field, disced_6d, and the outcomes in labels.OUTCOMES: spell_1_new_job (1 for the job type of the first new job after the survey) and appl_job (the number of applications to the job type).

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

occu_choice_pf.run_tests({'long': long})

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The frame is written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs.

The data tests (tests/test_regressions_data.py, marked data) check what the table assumes of its input but cannot check itself, and build both default tables:

• one row per person and job type;
• the outcomes, coefficients and controls are numbers or Booleans;
• T, the pre-treatment controls and the education columns take one value per person;
• T is 0 or 1, and T_vj_vij_job is T * vj_vij_job, null where vj_vij_job is;
• the outcomes are at least 0;
• both default tables build, keep T_vj_vij_job and vj_vij_job in every column, and count no more rows or persons than the frame has.

A failed data test shows the offending values. The other tests check the code on fake data; they give the same result anywhere, so on the server they test the environment.

Layout

src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       imported from the top: from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>

Every example also runs in the VS Code Interactive window, whole or cell by cell (# %%).

Adding a module

1. functions/<name>.py: importable at once as occu_choice_pf.<name>; arguments as {column: label} dicts, with defaults in labels.py
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), and split sections with # %%. tests/test_examples.py runs every example.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames.

Development

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames

The lower bounds in pyproject.toml are the server's versions for loguru, numpy and polars (as for ad-data-pf). To test against exactly those, in a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push (.github/workflows/test.yml).

Release

Bump version in pyproject.toml, run uv lock, and turn the ## Unreleased section of CHANGELOG.md into the version's entry. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag occu_choice_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/publish.yml):

git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0

The same tag runs .github/workflows/publish_1.yml, which publishes the same code as occu_choice_pf_1. The DST server caches a package when it is installed and won't update that name for 24 hours, so install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z

Both install the import occu_choice_pf, and the most recent install is the one used.

Before the first release, register both names as pending trusted publishers on pypi.org (Account settings, Publishing): owner AskerNC, repository occu_choice_pf, environment pypi, and workflow publish.yml for occu-choice-pf, publish_1.yml for occu_choice_pf_1.