Skip to main content

hdfe-stream

Regression with several high-dimensional fixed effects, on data that does not fit in memory: linear (feols_stream), and Poisson, logit and probit (fepois_stream, feglm_stream), with the Kline–Saggio–Sølvsten bias correction for AKM variance decompositions (leave_out_kss).

from hdfe_stream import feols_stream

fit = feols_stream(
    "log_earn ~ age_squared + age_cubed | worker_id + firm_id + year",
    "data/*.parquet",              # a Parquet path or glob, never fully loaded
    workdir="scratch",             # where intermediates go: disk, not memory
    vcov={"CRV1": "worker_id"},
)
fit.summary()

Formulas follow pyfixest/fixest syntax and are parsed by pyfixest's own parser, so the syntax, the coefficient names and the results all match. The difference is where the work happens: rows are streamed off disk in batches and never held as a design matrix.

The data can be a Parquet path or glob, a Polars LazyFrame, or a Polars DataFrame. Using file paths or LazyFrames keeps memory usage to a minimum, but using DataFrames works well for smaller data and for trying things out.

Documentation: leetucker.github.io/hdfe-stream, with examples, the KSS bias correction, benchmarks and the function reference.

Features

When to use this, and when not to

When your data do not fit in memory, use this. It was built for that case: a matched employer–employee panel — tens of millions of rows, millions of worker effects, an AKM variance decomposition — on a machine or a secure enclave where the data are larger than the memory you are allowed. Every other library compared here needs the whole panel in memory before it starts; this one holds a bounded slice of it at a time, and uses scratch disk for the rest.

When they do fit, it is still a sensible default for medium and large regressions. From about 400,000 workers (3.4 million rows) up, it was the fastest of the libraries benchmarked: against pyfixest, xhdfe, R's fixest and Julia's FixedEffectModels.jl on the AKM regression and on wide designs, against pyfixest on Poisson and logit, and against xhdfe on the KSS decomposition, and at the largest sizes it used a fraction of their memory. At 5 million workers it fitted the AKM regression in 22 seconds on a 16-vCPU machine, about as fast as xhdfe on an NVIDIA H100 (24 seconds), without a GPU (see should I use a GPU?). The benchmarks are simulated employer–employee panels, a shape that suits its design (how it works); one whose fixed-effect cells are no smaller than the data may do less well.

On small data it gains you nothing. Below about 400,000 workers the in-memory libraries were faster — by a second or less on the regression, and by six seconds on the smallest KSS decomposition — and lighter: an hdfe_stream process starts at about 0.8 GB.

Use pyfixest when you need what a streaming design does not offer. It is more mature and more widely used, and has inference and clustering options this library lacks or limits: CRV3 with fixed effects that are not nested in the clusters, separation checks on the covariates and bias corrections for Poisson and logit, and more (see limitations). Use xhdfe when GPU speed is the priority: its CUDA backend made the most of the GPU in the GPU benchmark.

This library is at its strongest when:

  • the data do not fit in memory, or fit so tightly that nothing else does;
  • you have one very high-cardinality dimension (workers, patients, students) whose groups each touch only a few levels of the others;
  • you want worker-specific slopes as well as intercepts (worker_id[t]), which multiplies the parameter count by the number of slopes;
  • the design is wide: hundreds of covariates, such as a categorical expanded into fine indicators or many interactions. An in-memory library needs rows × covariates in memory before it starts; this needs one batch of rows × covariates at a time, with any number of fixed effects, including one or none (see many covariates);
  • you need a hard, predictable memory ceiling — a shared cluster, or a job that must not be the one that gets killed.

You are trading memory for disk, and you need scratch space: about five times the size of your Parquet input while the fit runs, and only the result files once it finishes. See performance for what that buys. Point workdir= at it, or set the HDFE_STREAM_WORKDIR environment variable once (for example to a cluster's scratch filesystem) and leave workdir out; a workdir passed to a call takes precedence, and with neither, the system temporary directory is used.

How it works, in one paragraph

One fixed-effect dimension — normally the largest — is streamed: it is never represented as a vector in memory. Every other dimension is held as a vector sized by its number of levels, not rows. Rows are read in batches, hash-partitioned into buckets by the streamed dimension, sorted one bucket at a time, and reduced to a table of cells (one per combination of fixed-effect levels). The reduced normal equations for the non-streamed dimensions are then solved by conjugate gradient, and the streamed effects are recovered group by group in a final pass that writes residuals and fixed effects straight to Parquet. So memory scales with firms × years, not with workers × rows, and that asymmetry is the whole design. The covariates travel with the rows: what is held at once is one bucket or batch of rows × covariates, never all of them, which is why a wide design fits where an in-memory one does not. Full detail is in hdfe_stream/__init__.py.

Install

pip install hdfe-stream[formula]     # formula syntax needs pyfixest

polars, numpy, numba, pyarrow and scipy are required. The extras:

extra what it adds
formula feols_stream formula syntax and pyfixest reporting (pyfixest, formulaic, pandas)
within the solver="within" backend
amg algebraic multigrid preconditioning (precond="amg") for hard geometries
all all of the above
test the test suite
docs building the documentation (Sphinx)

On Python 3.15, pyfixest 0.60 (for the formula extra) has no wheels yet. It builds from source with a Rust toolchain and PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 set; everything else installs from wheels.

Without formula you still get StreamingHDFE, the lower-level interface that takes column names or Polars expressions instead of a formula string.

The experimental GPU options need CuPy and cudf-polars for your CUDA version (pip install cupy-cuda13x cudf-polars-cu13); there is no extra for them, since the packages differ by CUDA version. See CPU and GPU.

Performance

The standard three-way AKM regression, log_earn ~ age_squared + age_cubed | worker_id + firm_id + year, clustered by worker, on a simulated panel of 5 million workers and 42.5 million rows, against pyfixest, xhdfe, R's fixest and Julia's FixedEffectModels.jl. Every configuration agrees on the coefficients to within 2e-9.

configuration wall time peak memory peak disk
pyfixest (MAP, its default) 919 s 20.8 GB —
pyfixest (LSMR) 74 s 27.0 GB —
xhdfe 79 s 19.3 GB —
fixest (R) 152 s 9.1 GB —
FixedEffectModels.jl (Julia) 79 s 9.5 GB —
hdfe_stream (explicit, its default) 22 s 3.9 GB 2.6 GB
hdfe_stream (stream_cg) 22 s 3.9 GB 2.6 GB
hdfe_stream (stream_cg, low memory) 25 s 3.6 GB 2.7 GB
AKM regression: peak memory against the number of workers, one line per configuration, log-log

On small data the in-memory libraries use less memory: an hdfe_stream process costs about 0.8 GB before it has done anything. The benchmarks cover every size from 212,000 rows up, designs with up to 503 covariates, Poisson and logit, and the KSS decomposition, with how each was measured.

Should I use a GPU?

Probably not for this. hdfe_stream's GPU options are experimental, and on an NVIDIA H100 their gains were modest throughout and appeared only at large sizes: about 10% at 5 million workers with both options on, and no consistent gain below that. A GPU-oriented library such as xhdfe is the better choice if GPU performance is the priority: its CUDA backend took 24 s at 5 million workers, against 107 s on its CPU backend. But hdfe_stream on a 16-vCPU CPU machine took 22 s, and that machine costs about an eighth as much per hour as AWS's smallest H100 machine. CPU and GPU has the numbers.

Documentation

  • Examples: one page per feature, on simulated data that ships with the package: formulas and standard errors, weights and IV, Poisson and logit, wide designs, varying slopes, connected sets, running out of core, and reporting.
  • Leave-out variance components: the Kline–Saggio–Sølvsten bias correction for the AKM decomposition, its standard errors and weak-identification diagnostic, and every way it differs from the reference implementations.
  • Benchmarks: time, memory and disk against pyfixest, xhdfe, fixest and FixedEffectModels.jl.
  • Function reference: every option.
  • Limitations: what it does not do.
  • Changelog.

The scripts behind the examples are in examples/, and the benchmarks in benchmarks/.

Metadata

Release files for hdfe-stream 0.5.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hdfe-stream 0.5.1
File Size Uploaded Provenance Actions
hdfe_stream-0.5.1.tar.gz 482.1 kB With provenance Details for hdfe_stream-0.5.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hdfe-stream 0.5.1
File Interpreter ABI Platform Actions
hdfe_stream-0.5.1-py3-none-any.whl Python 3 none any Details for hdfe_stream-0.5.1-py3-none-any.whl

Total release size: 697.4 kB

Release files / hdfe_stream-0.5.1.tar.gz

Download URL hdfe_stream-0.5.1.tar.gz
Size 482.1 kB
Tags Source
SHA-256 checksum
How to use checksums
02bc8b7a346e297770baa776123f24b53bfe9cdaedaadf948823d35be89cedc5
BLAKE2b-256 checksum
How to use checksums
31c3e6ab6d89259bb71cbdef4d1244dd62fa212a6d9fd1216141d73d743561c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / hdfe_stream-0.5.1-py3-none-any.whl

Download URL hdfe_stream-0.5.1-py3-none-any.whl
Size 215.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a5b6a664d72273398a4124ddb488dcc25446d566385fde1573951c2366bcf52c
BLAKE2b-256 checksum
How to use checksums
7b5e6bb19d3cb8b36f7f73c0bad85c3a21bb5d27042729ccc88649dadd872a20
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for hdfe-stream 0.5.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for hdfe_stream-0.5.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page