occu-choice-pf

Tables for the survey experiment of Perceived Wage Differences and Early Career Job Search Choices. The functions are written and tested locally against simulated data, then run on the actual data. All numbers in this package come from simulated data.

Use

from occu_choice_pf import regressions as reg, save_tex, setup_log

setup_log(log_folder, 'regressions')      # logs the package version first
save_tex(reg.reg_control_table(long), out / 'tab_poisson.tex')               # Poisson
save_tex(reg.reg_control_table(long, method='ols'), out / 'tab_ols.tex')     # OLS

Tables return a bare tabular in the style of the paper's tables (the \sym stars of esttab included); save_tex writes it and logs it, so it can be read before an export request. Defaults (variables, labels) sit in occu_choice_pf.labels and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

The table of controls

reg_control_table(long) regresses each outcome on T_vj_vij_job, one panel per outcome, with one table column per entry of cols_control. Every column has coefs (reported), common_control, and the controls of its entry. The defaults (labels.COLS_CONTROL, labels.COMMON_CONTROL) give:

Column	Controls	cols_control entry
(1)	vj_vij_job and the person fixed effect (common_control)	{}
(2)	+ vj_vij_job squared and cubed	POWERS
(3)	+ pre-treatment controls times vj_vij_job	{**POWERS, **PRE_CONTROLS}
(4)	(3) + education field dummies times vj_vij_job	{**POWERS, **PRE_CONTROLS, **EDUC_FIELD}
(5)	(3) + DISCED (disced_6d) dummies times vj_vij_job	{**POWERS, **PRE_CONTROLS, **DISCED}

A control is {check-mark row label: terms}, and a term is a column ('age'), a column made into a dummy for each value ('i.age', as Stata's i.), or a polars expression (c.vj_vij_job ** 2). Your own columns:

from occu_choice_pf.labels import DISCED, POWERS, TIMES_MIS
own = {'Age dummies, gender ' + TIMES_MIS: ['i.age', 'female']}
reg.reg_control_table(long, cols_control=[{}, POWERS, {**POWERS, **own}, {**own, **DISCED}])

Every term of cols_control is times vj_vij_job, except one wrapped in reg.plain(), which enters its column as it is; an i.<column> there is a fixed effect absorbed in that column only:

from occu_choice_pf.labels import JOB
reg.reg_control_table(long, cols_control=[{}, {'Job type FE': reg.plain('i.' + JOB)}])

The arguments, the check-mark rows and the input checks are in help(reg.reg_control_table); the estimation, and what drops out of a column (all of it logged), in help(reg). reg.reg_control_fits(long, ...) returns the pyfixest fits behind a table, {outcome: [fit per column]}, to look into with .summary() or pf.etable. examples/ has custom calls: a subsample, other regressors, clustering and a second fixed effect.

The columns of a panel can rest on different rows: the Poisson regression with the DISCED dummies drops the separated rows of persons alone in their DISCED, which the other columns keep. common_sample=True estimates every column on the rows all of them keep, so that the columns differ in their controls only; reg.reg_control_samples(long, ...), with the arguments of the table, returns long with a Boolean column in_<outcome>_<column> per regression, True in the rows it is estimated on:

save_tex(reg.reg_control_table(long, common_sample=True), out / 'tab_poisson_common.tex')
s = reg.reg_control_samples(long)
s.filter(c.in_appl_job_5)                      # the rows of panel B, column (5)

The treated share within majors

With the major dummies times vj_vij_job (columns 4 and 5), the treatment effect is estimated within majors: a major whose persons are all treated, or all in the control group, adds next to nothing, and a major weighs more the closer its treated share is to one half. Without them, a treated share that differs between majors moves the estimate where the slope on vj_vij_job differs between majors. treatment_shares shows how large the persons' majors are and how far the treated share of their major is from what random assignment gives (T permuted across the persons, 1000 draws, no strata). Three table columns or a plot per column of labels.EDUC: major field and DISCED.

from occu_choice_pf import save_fig
from occu_choice_pf import treatment_shares as ts

save_tex(ts.percentile_table(long), out / 'tab_share_percentiles.tex')
save_fig(ts.share_figure(long), out / 'fig_share_hist.pdf')
ts.share_frame(long, 'disced_6d')     # one row per major: on the server only

• percentile_table: percentiles over persons of the size of their major and of its treated share (data and random assignment). Each percentile is the mean of the 9 persons around it (window), never one person's value. Below them, the standard deviation over persons of the treated share of their major (data and random assignment) and its p-value under random assignment: a permutation chi-squared test of treatment by major.
• share_figure: a histogram of the persons by the treated share of their major (0, ten intervals, 1), with the mean and the 95% range of random assignment; a matplotlib figure, the numbers plotted in the log. save_fig writes it.

The persons are those of the frame given, so the rows of a regression, s.filter(c.in_appl_job_5) with s = reg.reg_control_samples(long), give the shares within it.

Disclosure: no number shown rests on fewer than min_n (5) persons, nor can one be worked out from the others, the persons treated (in the paper's balance table) included. A count below 5, 0 included, reads <5. Percentiles are left empty where their 9-person windows overlap or leave 1-4 persons outside; the shares, the test and random assignment, which depends on the persons treated, with fewer than 5 treated or untreated. A bar of fewer than 5 persons reads <5; the bars at 0 and 1 are left out where the persons in all-treated or all-control majors, or the treated or untreated of the others, are 1-4; where the bars left out add up to 1-4 persons, the smallest interval bar is left out too. A 0 can still be worked out. share_frame lists every major, however small, and is for the server only.

The table of wage gap underestimation

underestimation_gap.gap_table(rows) is Table 3 of the paper: one row per subsample, rows={label: frame}, each frame the extra-long frame or a subsample of it, made with polars before the call.

from occu_choice_pf import underestimation_gap as ug

clean = {'d_vj_vij': c.d_vj_vij_clean, 'd_Vj_Vij_rate': c.d_Vj_Vij_rate_clean}
rows = {
    'Full sample': elong,
    **{f'Actual log wage gap $>$ {x}': elong.filter(c.d_vj > x) for x in [0.05, 0.1]},
    'Full sample, correcting reporting errors': elong.with_columns(**clean),
    'Reweighted for non-participation': elong,
}
weights = {'Reweighted for non-participation': 'weight_invited'}
save_tex(ug.gap_table(rows, weights), out / 'tab_estimation_gaps.tex')
save_tex(ug.gap_table(rows, weights, bootstrap=1000), out / 'tab_estimation_gaps_se.tex')

The rows are numbered. bootstrap=B adds standard errors from B draws of persons with replacement. compare=[(label, label)] puts the p-value of the difference between two rows under them ($p$-value of difference (1)-(2)), and a line under that. The rows need not be next to each other, and a row may be in several pairs: compare=[('Men', 'Women'), ('Men', 'Women, reweighted to (1)')] puts both p-values under the three rows (the rows from the first to the last of a pair are a block, joined with the blocks it overlaps). p_at_end=True puts all the p-values in the last rows of the table instead. share_column sets column 7: 'underestimator' (the default), the share of individuals who are underestimators; 'zero_gap', the share of job pairs with no perceived wage gap (d_Vij == 0); a list of both, ['underestimator', 'zero_gap'], columns 7 and 8 in that order; or None, no column 7. The frames need not have a column not shown. A frame with NaN raises, since a filter such as c.d_vj > 0.05 keeps NaN: make it missing in prep, elong = elong.with_columns(pl.col(pl.Float64).fill_nan(None)). ps_group reweights one group to the other with a propensity score model estimated anew in each draw, as Table C1 of the parental transmission paper: it adds ps_weight (reweighted to the other group) and ps_support (1 for persons with a weight) to every row, to name in weights:

no, yes = elong.filter(c.par_anyjobtype == 0), elong.filter(c.par_anyjobtype == 1)
rows = {'Neither': no, 'Either': yes, 'Neither, weightable': no, 'Either, reweighted to (3)': yes}
ug.gap_table(rows, {'Neither, weightable': 'ps_support', 'Either, reweighted to (3)': 'ps_weight'},
             bootstrap=1000, ps_group='par_anyjobtype',
             compare=[('Neither', 'Either'), ('Neither, weightable', 'Either, reweighted to (3)')])

The model is a logit of ps_group on ps_xs with the fixed effects ps_fes (majors, disced_6d, by default), in pyfixest; with one fixed effect and no ps_xs it is the group's share in each major. The cells, the weights and what is left out are in help(ug.gap_table) and help(ug.ps_weights).

The long frame

One row per person and job type (pnr x type_sizerank_job), with the columns of misc_docs/all_waves_long_variable_definitions.txt: T, vj_vij_job, T_vj_vij_job, the pre-treatment variables in labels.PRETREATMENT, educ_field, disced_6d, and the outcomes in labels.OUTCOMES: spell_1_new_job (1 for the job type of the first new job after the survey) and appl_job (the number of applications to the job type).

The extra-long frame

One row per person and ordered pair of job types (pnr x type_sizerank_job x type_sizerank_jobj), each pair in both orders; a d_ column is job j minus job j'. gap_table uses d_Vj_Vij_rate, d_vj_vij, d_v_vi_max_notmaxV_rate (one value per person) and, to reweight, disced_6d.

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

occu_choice_pf.run_tests({'long': long, 'elong': elong})

The data tests of a frame left out are skipped.

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The frame is written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs.

The data tests (tests/test_regressions_data.py, marked data) check what the table assumes of its input but cannot check itself, and build both default tables:

• one row per person and job type;
• the outcomes, coefficients and controls are numbers or Booleans;
• T, the pre-treatment controls and the education columns take one value per person;
• T is 0 or 1, and T_vj_vij_job is T * vj_vij_job, missing (null or NaN) where vj_vij_job is;
• the pre-treatment controls have no missing values, so that the columns of a panel rest on the same rows but for what the Poisson regression drops;
• the outcomes are at least 0;
• both default tables build and keep T_vj_vij_job and vj_vij_job in every column, and pyfixest's estimation sample, whose persons the Individuals row and min_n count, has the fit's N rows and carries the row numbers that common_sample and reg_control_samples read (a check on the server's pyfixest version).

tests/test_treatment_shares_data.py checks that the education columns are there and builds the exhibits of treatment_shares.

A failed data test shows the offending values. The other tests check the code on fake data; they give the same result anywhere, so on the server they test the environment.

Layout

src/occu_choice_pf/
├── functions/          one module per topic: regressions.py, labels.py, utils.py
│                       underestimation_gap.py, treatment_shares.py; imported from the top:
│                       from occu_choice_pf import regressions
├── simulation/         fake data per module (regressions.py) and the files in data/
├── examples/
│   └── regressions/    code/ runs every table on the fake data;
│                       logs/ and output/ hold what it writes
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import occu_choice_pf, pathlib; print(pathlib.Path(occu_choice_pf.__file__).parent)"
python <that folder>/examples/regressions/code/run_regressions.py <output folder>

Every example also runs in the VS Code Interactive window, whole or cell by cell (# %%).

Adding a module

1. functions/<name>.py: importable at once as occu_choice_pf.<name>; arguments as {column: label} dicts, with defaults in labels.py
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), and split sections with # %%. tests/test_examples.py runs every example.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames.

Development

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/occu_choice_pf/simulation/regressions.py   # save the fake data to simulation/data
uv run pytest --data src/occu_choice_pf/simulation/data      # data tests on saved frames

The lower bounds in pyproject.toml are the server's versions for loguru, numpy and polars (as for ad-data-pf). To test against exactly those, in a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push (.github/workflows/test.yml).

Release

Bump version in pyproject.toml, run uv lock, and turn the ## Unreleased section of CHANGELOG.md into the version's entry. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag occu_choice_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/publish.yml):

git tag occu_choice_pf-v0.1.0 && git push origin occu_choice_pf-v0.1.0

The same tag runs .github/workflows/publish_1.yml and publish_2.yml, which publish the same code as occu_choice_pf_1 and occu_choice_pf_2. The DST server caches a package when it is installed and won't update that name for 24 hours, so install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install occu-choice-pf==X.Y.Z
pip install occu_choice_pf_1==X.Y.Z
pip install occu_choice_pf_2==X.Y.Z

All three install the import occu_choice_pf, and the most recent install is the one used.

Before a name's first release, register it as a pending trusted publisher on pypi.org (Account settings, Publishing): owner AskerNC, repository occu_choice_pf, environment pypi, and workflow publish.yml for occu-choice-pf, publish_1.yml for occu_choice_pf_1, publish_2.yml for occu_choice_pf_2.