binsmooth

Python implementation of "Better Estimates from Binned Income Data"

Better Estimates from Binned Income Data: Interpolated CDFs and Mean-Matching
Paul T. von Hippel, David J. Hunter, McKalie Drown
Sociological Science
Volume 4, Number 26, Pages 641-655
2017

Originally implemented in the R package binsmooth.

Usage

from binsmooth import BinSmooth

bin_edges = np.array([0, 18200, 37000, 87000, 180000])
counts = np.array([0, 7527, 13797, 75481, 50646, 803])
mean_estimate = 95000

bs = BinSmooth()
bs.fit(bin_edges, counts, m=mean_estimate)

# Print median estimate
print(bs.inv_cdf(0.5))

Installation

Install via pip

pip install binsmooth

pypi page https://pypi.org/project/binsmooth/

Improvements

Better tail estimate by bounded optimisation rather than the adhoc search method found in the R implementation.

More precise inverse CDF by dynamically sampling the CDF. This is done by sampling proportional to the steepness of the CDF i.e. sampling more in areas where the CDF is steeper.

Warnings

Results may not exactly match R binsmooth because of a different approach to estimating the tail (upper bound).

Furthermore the fit method uses spline_type="PCHIP" by default, which is scipy's PchipInterpolator [1]. While the R implementation uses the spline from [2], which can be mimicked by setting spline_type="HYMAN".

References

[1]: Fritsch, F. N. and Carlson, R. E. (1980). Monotone piecewise cubic interpolation. SIAM Journal on Numerical Analysis
[2]: Hyman, J. M. (1983). Accurate monotonicity preserving cubic interpolation. SIAM Journal on Scientific and Statistical Computing