snapparquet 0.1.1

Fast column-pruned Parquet reads with optional predicate pushdown for analytics.

snapparquet

Small, dependency-light helpers for fast Parquet workflows in Python: column pruning (read only the fields you need), optional integer-column range filters with PyArrow dataset predicate pushdown when row-group statistics allow, and footer-only schema reads for discovery.

Install

pip install snapparquet

From a checkout:

pip install -e .

Quick start

import snapparquet as sp

cols = ["user_id", "load_date", "revenue"]

df = sp.read_tabular(
    "metrics.parquet",
    columns=cols,
    filter_column="load_date",
    range_start=20250301,
    range_end=20250331,
)

# Many daily files
df = sp.read_tabular_many(
    ["day1.parquet", "day2.parquet"],
    columns=cols,
    filter_column="load_date",
    range_start=20250301,
    range_end=20250331,
)

API

Function	Purpose
read_parquet(path, columns=..., filter_column=..., range_start=..., range_end=...)	Parquet → pandas with pruning and optional range filter
read_csv(...)	CSV with usecols + same range filter (in memory)
read_tabular(path, ...)	Chooses Parquet vs CSV from the file suffix
read_tabular_many(paths, ...)	Concatenate multiple files
schema_column_names(path)	Column set from Parquet metadata only
intersect_columns(path, wanted)	Ordered intersection for columns=
column_is_integer_type(path, name)	Whether range pushdown may apply
schema_matches_any_group(names, groups)	Schema shape checks (e.g. alternate mart layouts)

When pushdown applies

If filter_column is stored as an integer type in the Parquet file, read_parquet / read_tabular try pyarrow.dataset first so row groups can be skipped using min/max statistics. Otherwise the same range filter runs in pandas after a (possibly pruned) read.

Publishing to PyPI

Follow the PyPA packaging tutorial: use TestPyPI first, then production PyPI.

1. Register at pypi.org and create an API token under Account settings → API tokens.
2. Confirm the project name is available: pip index versions snapparquet
3. From this directory:

python3 -m pip install --upgrade pip build twine
python3 -m build
python3 -m twine check dist/*

4. TestPyPI (recommended first upload):

python3 -m twine upload --repository testpypi dist/*

5. Production PyPI (omit --repository; default is pypi.org):

python3 -m twine upload dist/*

Use username __token__ and password = your token (including the pypi- prefix), or set TWINE_USERNAME=__token__ and TWINE_PASSWORD.

After publishing, set Repository / Issues in [project.urls] in pyproject.toml to your real Git host.

License

MIT