occu-choice-pf

Tables for the survey experiment of Perceived Wage Differences and Early Career Job Search Choices. The functions are written and tested locally against simulated data, then run on the actual data. All numbers in this package come from simulated data.

Use

from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS

Tables return a bare tabular in the style of the paper's tables (the \sym stars of esttab included); save_tex writes it and logs it, so it can be read before an export request. Defaults (variables, labels) sit in occu_choice_pf.labels and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

The table of controls

reg_control_table(long) regresses each outcome on T_vj_vij_job, one panel per outcome, with one table column per entry of cols_control. Every column has coefs (reported), common_control, and the controls of its entry. The defaults (labels.COLS_CONTROL, labels.COMMON_CONTROL) give:

Column	Controls	cols_control entry
(1)	vj_vij_job and the person fixed effect (common_control)	{}
(2)	+ vj_vij_job squared and cubed	POWERS
(3)	+ pre-treatment controls times vj_vij_job	{**POWERS, **PRE_CONTROLS}
(4)	(3) + education field dummies times vj_vij_job	{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}
(5)	(3) + DISCED (disced_6d) dummies times vj_vij_job	{**POWERS, **PRE_CONTROLS, **DISCED}

A control is {check-mark row label: terms}, and a term is a column ('age'), a column made into a dummy for each value ('i.age', as Stata's i.), or a polars expression (c.vj_vij_job ** 2). Your own columns:

from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])

Every term of cols_control is times vj_vij_job, except one wrapped in reg.plain(), which enters its column as it is; an i.<column> there is a fixed effect absorbed in that column only:

from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])

The arguments, the check-mark rows and the input checks are in help(reg.reg_control_table); the estimation, and what drops out of a column (all of it logged), in help(reg). reg.reg_control_fits(long, ...) returns the pyfixest fits behind a table, {outcome: [fit per column]}, to look into with .summary() or pf.etable. examples/ has custom calls: a subsample, other regressors, clustering and a second fixed effect.

The table of wage gap underestimation

underestimation_gap.gap_table(rows) is Table 3 of the paper: one row per subsample, rows={label: frame}, each frame the extra-long frame or a subsample of it, made with polars before the call.

from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')

The rows are numbered. bootstrap=B adds standard errors from B draws of persons with replacement. compare=[(label, label)] puts the p-value of the difference between two rows under them ($p$-value of difference (1)-(2)), and a line under that. The rows need not be next to each other, and a row may be in several pairs: compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')] puts both p-values under the three rows (the rows from the first to the last of a pair are a block, joined with the blocks it overlaps). p_at_end=True puts all the p-values in the last rows of the table instead. share_column sets column 7: 'underestimator' (the default), the share of individuals who are underestimators; 'zero_gap', the share of job pairs with no perceived wage gap (d_Vij == 0); a list of both, ['underestimator', 'zero_gap'], columns 7 and 8 in that order; or None, no column 7. The frames need not have a column not shown. A frame with NaN raises, since a filter such as c.d_vj > 0.05 keeps NaN: make it missing in prep, elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None)). ps_group reweights one group to the other with a propensity score model estimated anew in each draw, as Table C1 of the parental transmission paper: it adds ps_weight (reweighted to the other group) and ps_support (1 for persons with a weight) to every row, to name in weights:

no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])

The model is a logit of ps_group on ps_xs with the fixed effects ps_fes (majors, disced_6d, by default), in pyfixest; with one fixed effect and no ps_xs it is the group's share in each major. The cells, the weights and what is left out are in help(ug.gap_table) and help(ug.ps_weights).

The long frame

One row per person and job type (pnr x type_sizerank_job), with the columns of misc_docs/all_waves_long_variable_definitions.txt: T, vj_vij_job, T_vj_vij_job, the pre-treatment variables in labels.PRETREATMENT, educ_field, disced_6d, and the outcomes in labels.OUTCOMES: spell_1_new_job (1 for the job type of the first new job after the survey) and appl_job (the number of applications to the job type).

The extra-long frame

One row per person and ordered pair of job types (pnr x type_sizerank_job x type_sizerank_jobj), each pair in both orders; a d_ column is job j minus job j'. gap_table uses d_Vj_Vij_rate, d_vj_vij, d_v_vi_max_notmaxV_rate (one value per person) and, to reweight, disced_6d.

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

occu_choice_pf.run_tests({'long': long, 'elong': elong})

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The frame is written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs.

The data tests (tests/test_regressions_data.py, marked data) check what the table assumes of its input but cannot check itself, and build both default tables:

• one row per person and job type;
• the outcomes, coefficients and controls are numbers or Booleans;
• T, the pre-treatment controls and the education columns take one value per person;
• T is 0 or 1, and T_vj_vij_job is T * vj_vij_job, missing (null or NaN) where vj_vij_job is;
• the pre-treatment controls have no missing values, so that the columns of a panel rest on the same rows but for what the Poisson regression drops;
• the outcomes are at least 0;
• both default tables build and keep T_vj_vij_job and vj_vij_job in every column, and pyfixest's estimation sample, whose persons the Individuals row and min_n count, has the fit's N rows (a check on the server's pyfixest version).

A failed data test shows the offending values. The other tests check the code on fake data; they give the same result anywhere, so on the server they test the environment.

Layout

src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py; imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>

Every example also runs in the VS Code Interactive window, whole or cell by cell (# %%).

Adding a module

1. functions/<name>.py: importable at once as occu_choice_pf.<name>; arguments as {column: label} dicts, with defaults in labels.py
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), and split sections with # %%. tests/test_examples.py runs every example.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames.

Development

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames

The lower bounds in pyproject.toml are the server's versions for loguru, numpy and polars (as for ad-data-pf). To test against exactly those, in a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push (.github/workflows/test.yml).

Release

Bump version in pyproject.toml, run uv lock, and turn the ## Unreleased section of CHANGELOG.md into the version's entry. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag occu_choice_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/publish.yml):

git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0

The same tag runs .github/workflows/publish_1.yml and publish_2.yml, which publish the same code as occu_choice_pf_1 and occu_choice_pf_2. The DST server caches a package when it is installed and won't update that name for 24 hours, so install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z

All three install the import occu_choice_pf, and the most recent install is the one used.

Before a name's first release, register it as a pending trusted publisher on pypi.org (Account settings, Publishing): owner AskerNC, repository occu_choice_pf, environment pypi, and workflow publish.yml for occu-choice-pf, publish_1.yml for occu_choice_pf_1, publish_2.yml for occu_choice_pf_2.