parquet-file-fs

ls, cat and glob the files stored inside parquet shards — a read-only fsspec filesystem with a Rust core.

Read-only fsspec filesystem over parquet "archive" shards — parquet files where each row is one file (a path column + a content column). Rust core (arrow/parquet + object_store) via maturin; built for handing datasets to coding agents: ls, cat, open, glob, walk over shard contents without loading whole shards into memory.

Install

pip install parquet-file-fs

Usage

from parquet_file_fs import ParquetFileSystem

fs = ParquetFileSystem("data/*.parquet")            # local glob
fs = ParquetFileSystem("s3://bucket/shard-*.parquet")  # S3 (env credentials)
fs = ParquetFileSystem(["a.parquet", "b.parquet"])  # multi-shard

fs.ls("images", detail=False)     # ['images/a.png', 'images/b.png']
fs.cat_file("labels/a.json")      # b'{"route": "agentic"}'
fs.glob("**/*.json")              # index-only, no content decoding
fs.info("images/a.png")           # {'name': ..., 'size': 5, 'type': 'file',
                                  #  'metadata': {<extra parquet columns>}}
with fs.open("readme.txt", "r") as f:
    print(f.read())

Column names are auto-detected (path/filename/file_name/key and content/data/bytes) or set explicitly:

fs = ParquetFileSystem("odd.parquet", path_column="file_name",
                       content_column="image_bytes")

Duplicate paths across shards raise by default; pass on_duplicate="first" or "last" to pick a shard instead.

Protocol adapters

Built-in: local paths / file://, s3:// (credentials from the environment), http://, https://. Any other scheme can be registered — either an fsspec filesystem or a tiny duck-typed object:

import gcsfs
from parquet_file_fs import ParquetFileSystem, register_adapter

register_adapter("gs", gcsfs.GCSFileSystem())
fs = ParquetFileSystem("gs://bucket/data/*.parquet")

Custom adapter interface: size(url) -> int, read_range(url, offset, length) -> bytes (must return exactly length bytes; callers never request past EOF), glob(pattern) -> list[str].

Glob patterns are supported for local paths and s3://; pass concrete URLs for http(s)://.

Notes

• Read-only: mutation methods raise NotImplementedError.
• The index is built from parquet footers + the path column only; file content decodes lazily per row group (small LRU cache). Exact file sizes are computed on first info/ls(detail=True) per row group.
• Registered with fsspec as protocol pfs.

Development

uv sync                 # dev deps from [dependency-groups]
uv run maturin develop  # build + install the extension
uv run pytest           # Python tests
cargo test              # Rust tests
cargo clippy --all-targets -- -D warnings
cargo fmt --all

Without uv:

python3 -m venv .venv && .venv/bin/pip install maturin pytest pyarrow pyyaml fsspec
.venv/bin/maturin develop && .venv/bin/pytest

Releasing

Commits on master drive release-please: it opens a release PR that bumps pyproject.toml, Cargo.toml (via the x-release-please-version marker) and Cargo.lock. Merging that PR tags the release and publishes wheels (linux x86_64, macOS arm64) plus an sdist to PyPI through Trusted Publishing — no API tokens in the repo. A failed publish can be re-run with workflow_dispatch on the release-please workflow — that path also publishes the current master version directly, which is the way out if the release PR itself can't be opened.

Opening that PR requires Allow GitHub Actions to create and approve pull requests, enabled at both the repo (Settings → Actions → General) and the account level (github.com/settings/actions). Where that policy is unavailable, add a PAT with contents: write + pull-requests: write as the RELEASE_PLEASE_TOKEN secret and the workflow uses it instead.

License

MIT OR Apache-2.0, at your option.