logit-stream

Logit models for data that do not fit in memory: binary, multinomial, ordered, conditional, nested and mixed logit, fitted by Newton's method with the data streamed off disk, from pyfixest-style formulas, with absorbed fixed effects or random intercepts.

from logit_stream import mlogit_stream

fit = mlogit_stream(
    "race ~ C(agecat) + female + C(inccat) + C(cit) | puma",
    "acs/*.parquet",               # a Parquet path or glob, never fully loaded
    vcov={"CRV1": "puma"},
)
fit.summary()
fit.predict("people/*.parquet").sink_parquet("imputed.parquet")   # lazily

Newton's method for a logit needs the data only through sums over rows -- the log-likelihood, its gradient and the information matrix -- so each step is one read of the data, block by block, and one dense solve; the likelihoods are concave, so a handful of steps suffice. Memory grows with the number of coefficients, not of rows. The multinomial and ordered logit absorb one fixed effect exactly, by eliminating its per-level blocks of the information matrix in every step.

Results share hdfe-stream's reporting: summary(), tidy(), summary_dict(), summary_json(), confint(), wald_test(), with_vcov(), etable(), and lazy predict(), resid(), sample() and fixef() frames. Estimates and standard errors match statsmodels and scikit-learn (see tests/).

Documentation: https://leetucker.github.io/logit-stream/ (build locally with pip install -e ".[docs]" and sphinx-build -b html docs docs/_build/html). examples/ has a runnable script for each page of its guide, on simulated data.

Install

pip install logit-stream

Python 3.10 or later. On Python 3.15, pyfixest 0.60 (which hdfe-stream's formula extra brings in) has no wheels yet: it builds from source with a Rust toolchain and PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 set; everything else installs from wheels.

Benchmarks

benchmarks/ compares logit-stream with scikit-learn, statsmodels, mord and R's glm, fixest, nnet, MASS and ordinal on American Community Survey microdata (10,000 to 12.9 million adults); with xlogit, pylogit, statsmodels, jaxlogit, Biogeme and R's mlogit and logitr on the Swissmetro choice survey, resampled to 10 million choice sets; on short panels with statsmodels and R's survival, fixest and alpaca; and on a GPU. See benchmarks/README.md and the documentation's benchmark pages.

License

MIT.