Survey Kit

Tools for addressing missing data problems (nonresponse bias and item missingness) including extremely fast calibration weighting and machine learning-based imputation.

A furlough project inspired by the code used for the U.S. Census Bureau for the National Experimental Wellbeing Statistics (NEWS) project.

Installation

uv add survey-kit
# or
pip install survey-kit

Running regressions with multiply imputed data (below) can be easier with the tools noted below, but they are not added as dependencies by default. Add only what you use:

uv add "survey-kit[pyfixest]"    # or [statsmodels], [linearmodels], [polars-ds], [r], [stata]
uv add "survey-kit[all-stats]"   # everything at once

For R and Stata, there's some additional setup you need to do to get them working with python. See rpy2's installation instructions and pystata's documentation.

Older Linux (e.g., RHEL 7)

Many dependencies (mkl, pyarrow, lightgbm, numba, igraph, ...) have stopped publishing wheels for old glibc versions (< 2.28). Use the old-linux extra, which pins the last versions that still have compatible wheels:

uv add "survey-kit[old-linux]"

Features

• Calibration Weighting - Fast entropy balancing for nonresponse bias
• SRMI Imputation - ML-based multiple imputation with checkpointing
• Statistics & Standard Errors - Proper variance estimation for complex surveys
• Running regressions with multiply imputed data using standard python, R, or Stata tools - statsmodels, linearmodels, pyfixest, R, and Stata behind one interface

Works with Polars, Pandas, Arrow, and DuckDB.

Documentation

Full documentation: https://jrothbaum.github.io/survey_kit/

• Calibration Guide
• Imputation Guide
• Statistics Guide
• Guide to running regressions with multiply imputed data using standard python, R, or Stata tools

Support

• Issues

License

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.