snapparquet 0.1.2

Small helpers for fast Parquet/CSV: column pruning, optional integer range filters with PyArrow pushdown, footer-only schema.

snapparquet

Project description

snapparquet provides small, dependency-light helpers for fast Parquet workflows in Python: column pruning (read only the fields you need), optional integer-column range filters with PyArrow dataset predicate pushdown when row-group statistics allow, and footer-only schema reads for discovery. It fits analytics pipelines and dashboards that aggregate many daily Parquet (or CSV) files.

Install

pip install snapparquet

From a checkout:

pip install -e .

Quick start

import snapparquet as sp

cols = ["user_id", "load_date", "revenue"]

df = sp.read_tabular(
    "metrics.parquet",
    columns=cols,
    filter_column="load_date",
    range_start=20250301,
    range_end=20250331,
)

# Many daily files
df = sp.read_tabular_many(
    ["day1.parquet", "day2.parquet"],
    columns=cols,
    filter_column="load_date",
    range_start=20250301,
    range_end=20250331,
)

API

Function	Purpose
read_parquet(path, columns=..., filter_column=..., range_start=..., range_end=...)	Parquet → pandas with pruning and optional range filter
read_csv(...)	CSV with usecols + same range filter (in memory)
read_tabular(path, ...)	Chooses Parquet vs CSV from the file suffix
read_tabular_many(paths, ...)	Concatenate multiple files
read_parquet_many(paths, ...)	Same as read_tabular_many (alias for Parquet-heavy workflows)
schema_column_names(path)	Column set from Parquet metadata only
intersect_columns(path, wanted)	Ordered intersection for columns=
column_is_integer_type(path, name)	Whether range pushdown may apply
schema_matches_any_group(names, groups)	Schema shape checks (e.g. alternate mart layouts)

Also exported: norm_col, schema_has_all.

When pushdown applies

If filter_column is stored as an integer type in the Parquet file, read_parquet / read_tabular try pyarrow.dataset first so row groups can be skipped using min/max statistics. Otherwise the same range filter runs in pandas after a (possibly pruned) read.