microdf
Weighted pandas DataFrames and Series for survey microdata analysis.
Why this exists
Survey microdata comes with weights, and analysing it in pandas means getting two things right that pandas will not do for you.
The estimators are not the obvious ones. A weighted median is not the median of
weighted values. Weighted variance requires choosing between treating weights as
frequencies or as precision. A top-1% share requires deciding what happens to a
record that straddles the cutoff. Each of these is a decision, and hand-rolling it
per analysis means making it differently each time. microdf makes each choice
once, documents it, and tests it — quantiles follow the inverse CDF so they can be
checked against R's survey::svyquantile, and variance treats weights as
frequencies so integer weights agree with numpy on the replicated sample.
Weights have to survive the pipeline. Before any estimator runs, weights must
stay aligned with their rows through merges, filters, grouping and reindexing. When
they do not, nothing raises. The pipeline completes and returns a plausible wrong
number. This is the harder of the two problems, and it is why microdf carries
weights inside the object rather than beside it.
If you are computing a poverty rate or a Gini on weighted survey data, those are the two ways to get a believable-looking wrong answer.
Key Features
- MicroDataFrame: A pandas DataFrame with an integrated weight column
- MicroSeries: A pandas Series with integrated weights
- Weighted operations: All aggregations (sum, mean, median, etc.) automatically use weights
- Inequality metrics: Built-in Gini coefficient calculation
- Poverty analysis: Integrated poverty rate and gap calculations
Installation
Install with:
pip install microdf-python
Or for development:
pip install git+https://github.com/PolicyEngine/microdf.git
Usage
import microdf as mdf
import pandas as pd
# Create sample data with weights
df = pd.DataFrame(
{"income": [10_000, 20_000, 30_000, 40_000, 50_000], "weights": [1, 2, 3, 2, 1]}
)
# Create a MicroDataFrame
mdf_df = mdf.MicroDataFrame(df, weights="weights")
# All operations are weight-aware
print(mdf_df.income.mean()) # Weighted mean
print(mdf_df.income.gini()) # Gini coefficient
Questions
Contact the maintainer, Max Ghenis (max@policyengine.org).
Citation
You may cite the source of your analysis as "microdf release #.#.#, author's calculations."
2 maintainers
Metadata
2 maintainers
Release files for microdf-python 1.5.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| microdf_python-1.5.9.tar.gz | 67.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| microdf_python-1.5.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 143.8 kB
Release files / microdf_python-1.5.9.tar.gz
| Download URL | microdf_python-1.5.9.tar.gz |
|---|---|
| Size | 67.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
01ce7505ad4ae35372d129d2b4c1835b1aac2e7bc683ff9ae247c817878cf11a
|
|
BLAKE2b-256 checksum How to use checksums |
69877f97bde845c3cd10c1fe057d2e28f3c0517126213ec81dd0827f9da4697e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / microdf_python-1.5.9-py3-none-any.whl
| Download URL | microdf_python-1.5.9-py3-none-any.whl |
|---|---|
| Size | 75.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f5407720e936dd7ee52c1c44a75d7d7a39d407bcc2b4cd898fef73602d0615b8
|
|
BLAKE2b-256 checksum How to use checksums |
a6e45961a3f638c1003d67a3445334d8873f9a8a6a91b795f412845cd7c73f45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
