polars-io 0.3.0

Lazy and eager reading of Stata and SAS files into Polars

polars_io

Lazily read Stata (.dta), SAS (.sas7bdat, .xpt), and fixed-width (.txt, .dat, etc.) files in polars.

Installation

pip install polars_io # or uv add polars_io

Usage

import polars as pl
import polars_io as pio

# lazily load a sas file
lf = pio.scan_sas7bdat("huge_SAS_file.sas7bdat")

# get its schema
lf.collect_schema()

# take a look at the first few rows
lf.head().collect()

# projection and predicate pushdown works!
(
    lf
    .filter(pl.col("birth_year").is_between(2000, 2010))
    .select(pl.col("usage").mean())
    .collect()
)

# load fixed-width files
col_locations = { "year" : (10, 14), "population" : (14, 20) }
pio.scan_fwf("populations.txt", col_locations)

# eager versions of all functions are also available
pio.read_dta("mortality_rates.dta")

See the documentation for more info.

Details

The Stata and SAS implementations make use the readstat C library via the python bindings provided by pyreadstat. For numeric types, reading uses zero-copy conversions from numpy -> pyarrow -> polars and should be faster and have lower memory overhead than reading the data into pandas and then calling pl.from_pandas (benchmarks welcome).

Contributing

PRs adding support for reading other formats are very welcome! (E.g. .Rdata, Stata .dct, etc.)

Issues

This packages fails to some read files with non-utf8 metadata (e.g. column labels, notes on .dta files). This is a known issue with upstream packages that is being worked on (see Roche/pyreadstat#298 and WizardMac/ReadStat#344).