twowayfeweights (Python)

Estimates the weights and the sensitivity measures attached to two-way fixed effects (TWFE) and first-difference regressions, following de Chaisemartin & D'Haultfoeuille (2020). This is a Python port of the Stata and R command twowayfeweights. It reproduces the Stata results: see Parity with Stata.

Documentation: https://credible-answers.github.io/py_twowayfeweights/

A TWFE coefficient estimates a weighted sum of the treatment effects in each (g,t) cell. Some of those weights can be negative. When they are, the coefficient can have the opposite sign of every individual treatment effect. twowayfeweights computes these weights, reports how many are negative and how much they sum to, and measures how robust the coefficient is to heterogeneous effects.

Installation

pip install twowayfeweights

Requires Python ≥ 3.9. The only dependencies are numpy, pandas and scipy.

Quick start

from twowayfeweights import twowayfeweights, load_wagepan

df = load_wagepan()
res = twowayfeweights(df, Y="lwage", G="nr", T="year", D="union",
                      type="feTR", summary_measures=True, test_random_weights="educ")
print(res)

Under the common trends assumption,
the TWFE coefficient beta, equal to 0.1066, estimates a weighted sum of 967 ATTs.
820 ATTs receive a positive weight, and 147 receive a negative weight.
1016 (g,t) cells receive the treatment, but the ATTs of 49 cells receive a weight equal to zero.
------------------------------------------------
Treat. var: union       # ATTs      Σ weights
------------------------------------------------
Positive weights        820         1.0105
Negative weights        147         -0.0105
------------------------------------------------
Total                   967         1.0000
------------------------------------------------

Summary Measures:
TWFE coefficient (β_fe) = 0.1066
min σ(Δ) compatible with β_fe and Δ_TR = 0: 0.0969
min σ(Δ) compatible with treatment effect of opposite sign than β_fe in all (g,t) cells: 3.1759
Reference: Corollary 1, de Chaisemartin, C and D'Haultfoeuille, X (2020a)

Regression of variables possibly correlated with the treatment effect on the weights

B[1,4]
              Coef           SE       t-stat  Correlation
educ    -.13445527    .07136021   -1.8841771   -.11825874


The development of this package was funded by the European Union (ERC, REALLYCREDIBLE,GA N°101043899).

Usage

twowayfeweights(data, Y, G, T, D, type="feTR", D0=None, summary_measures=False,
                controls=None, weights=None, other_treatments=None,
                test_random_weights=None, path=None)

data must be a pandas DataFrame (convert other data frames first, e.g. df.to_pandas() for polars). Group and time identifiers can be numeric, strings or categoricals.

Estimation types

• feTR: regression with group and period fixed effects, under the common trends assumption.
• feS: the same regression, also assuming that each group's treatment effect does not change over time.
• fdTR: first-difference regression under common trends. Y and D are the first differences, and D0 is the treatment level.
• fdS: first-difference regression, also assuming stable treatment effects.

The result is a TwoWayFEWeightsResult. Its attributes follow the R package:

print(res) shows the Stata-style table. res.to_dict() returns the scalars. print_twowayfeweights(res) and dictionary access such as res["beta"] are kept for backward compatibility.

Example: first-difference regression with many controls

This reproduces chapter 5 of the DiD book, using data from Gentzkow, Shapiro & Sinkinson (2011):

styr = [c for c in df.columns if c.startswith("styr") and c != "styr"]   # 683 state-year dummies
res = twowayfeweights(df, "changeprestout", "cnty90", "year", "changedailies",
                      type="fdTR", D0="numdailies", controls=styr, test_random_weights="year")
# beta = 0.0026: 5371 positive weights (sum 2.4271), 4505 negative weights (sum -1.4271)

Parity with Stata

The test suite (tests/) checks 36 specifications (17 of them feS/fdS) against the Stata command from SSC. The specifications cover all four types, with and without controls, weights, other treatments and random-weight tests. They use the wagepan data, the Gentzkow et al. data with 683 controls, and a simulated unbalanced panel with gaps, several observations per cell and missing values. Stata's results are stored in tests/fixtures/stata/results.json (the command, every number it returns and the text it prints) and are regenerated by tests/reference/build_reference.py. The checks:

• the numbers of positive, negative and treated-cell weights match exactly;
• beta matches to 1e-7 relative;
• the weight sums and the sensitivity measures match to 1e-6 relative, the random-weights regression (Coef, SE, t-stat, Correlation) to 5e-6, and every (g,t) weight to 2e-6 of the largest weight.

python benchmarks/compare_r.py compares the results and run times with the R package (TwoWayFEWeights). Where R runs and agrees with Stata, Python matches R to 1e-8 or better.

The small remaining differences come from Stata. It stores intermediate variables (gen, predict, egen) as 4-byte floats, while this package computes everything in double precision.

One known difference: for type="feS" with several observations per (g,t) cell, Stata builds the running average of residuals one observation at a time, and then keeps an arbitrary observation of each cell. Its output then depends on the row order, and it changes from run to run because Stata breaks sort ties randomly. This package uses the formula from the paper: all observations in the periods t' ≥ t. With one observation per cell, which is the usual case, the results are identical.

Performance

Regressions run on (g,t) cells with summed weights, not on individual observations. The time dummies are never built as a dense matrix: after the group effects are absorbed, their normal equations come from a sparse group × period weight table. The system is solved with a scaled pseudo-inverse, which handles collinearity, plus iterative refinement. Everything else is vectorized numpy code.

Seconds per call on the same data, on one Windows laptop (Stata 18 MP, R 4.5 with TwoWayFEWeights 2.1.0):

python benchmarks/bench.py times panels of up to 3M rows.

References

• de Chaisemartin, C. and D'Haultfoeuille, X. (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. American Economic Review, 110(9), 2964–2996.
• de Chaisemartin, C. and D'Haultfoeuille, X. (2023). Two-way fixed effects and differences-in-differences estimators with several treatments. Journal of Econometrics.

Stata: ssc install twowayfeweights. R: install.packages("TwoWayFEWeights").

The development of the original package was funded by the European Union (ERC, REALLYCREDIBLE, GA N°101043899).