parquet-file-fs

parquet-file-fs

ls, cat and glob the files stored inside parquet shards — a read-only fsspec filesystem with a Rust core.

CI python-ci PyPI License

Read-only fsspec filesystem over parquet "archive" shards — parquet files where each row is one file (a path column + a content column). Rust core (arrow/parquet + object_store) via maturin; built for handing datasets to coding agents: ls, cat, open, glob, walk over shard contents without loading whole shards into memory.

Install

pip install parquet-file-fs

Usage

from parquet_file_fs import ParquetFileSystem

fs = ParquetFileSystem("data/*.parquet")            # local glob
fs = ParquetFileSystem("s3://bucket/shard-*.parquet")  # S3 (env credentials)
fs = ParquetFileSystem(["a.parquet", "b.parquet"])  # multi-shard

fs.ls("images", detail=False)     # ['images/a.png', 'images/b.png']
fs.cat_file("labels/a.json")      # b'{"route": "agentic"}'
fs.glob("**/*.json")              # index-only, no content decoding
fs.info("images/a.png")           # {'name': ..., 'size': 5, 'type': 'file',
                                  #  'metadata': {<extra parquet columns>}}
with fs.open("readme.txt", "r") as f:
    print(f.read())

Column names are auto-detected (path/filename/file_name/key and content/data/bytes) or set explicitly:

fs = ParquetFileSystem("odd.parquet", path_column="file_name",
                       content_column="image_bytes")

Duplicate paths across shards raise by default; pass on_duplicate="first" or "last" to pick a shard instead.

Protocol adapters

Built-in: local paths / file://, s3:// (credentials from the environment), http://, https://. Any other scheme can be registered — either an fsspec filesystem or a tiny duck-typed object:

import gcsfs
from parquet_file_fs import ParquetFileSystem, register_adapter

register_adapter("gs", gcsfs.GCSFileSystem())
fs = ParquetFileSystem("gs://bucket/data/*.parquet")

Custom adapter interface: size(url) -> int, read_range(url, offset, length) -> bytes (must return exactly length bytes; callers never request past EOF), glob(pattern) -> list[str].

Glob patterns are supported for local paths and s3://; pass concrete URLs for http(s)://.

Notes

  • Read-only: mutation methods raise NotImplementedError.
  • The index is built from parquet footers + the path column only; file content decodes lazily per row group (small LRU cache). Exact file sizes are computed on first info/ls(detail=True) per row group.
  • Registered with fsspec as protocol pfs.

Development

uv sync                 # dev deps from [dependency-groups]
uv run maturin develop  # build + install the extension
uv run pytest           # Python tests
cargo test              # Rust tests
cargo clippy --all-targets -- -D warnings
cargo fmt --all

Without uv:

python3 -m venv .venv && .venv/bin/pip install maturin pytest pyarrow pyyaml fsspec
.venv/bin/maturin develop && .venv/bin/pytest

Releasing

Commits on master drive release-please: it opens a release PR that bumps pyproject.toml, Cargo.toml (via the x-release-please-version marker) and Cargo.lock. Merging that PR tags the release and publishes wheels (linux x86_64, macOS arm64) plus an sdist to PyPI through Trusted Publishing — no API tokens in the repo. A failed publish can be re-run with workflow_dispatch on the release-please workflow — that path also publishes the current master version directly, which is the way out if the release PR itself can't be opened.

Opening that PR requires Allow GitHub Actions to create and approve pull requests, enabled at both the repo (Settings → Actions → General) and the account level (github.com/settings/actions). Where that policy is unavailable, add a PAT with contents: write + pull-requests: write as the RELEASE_PLEASE_TOKEN secret and the workflow uses it instead.

License

MIT OR Apache-2.0, at your option.

Project links

Data verified by PyPI on Jul 24, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ransomware from gravatar.com ransomware

Credits

Author: 4thel00z

GitHub Statistics

Data verified by PyPI on Jul 24, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT OR Apache-2.0
View SPDX License List

Requires

Python >=3.9

Tags

parquet fsspec filesystem virtual-filesystem arrow columnar dataset s3 pyo3 rust

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parquet_file_fs-0.2.0.tar.gz (76.7 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

parquet_file_fs-0.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.2 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

parquet_file_fs-0.2.0-cp39-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

File details

Details for the file parquet_file_fs-0.2.0.tar.gz.

File metadata

  • Download URL: parquet_file_fs-0.2.0.tar.gz
  • Upload date:
  • Size: 76.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for parquet_file_fs-0.2.0.tar.gz
Algorithm Hash digest
SHA256 4ed880e4e4c44b4347a5b6876155674d8581fb3663e134b1601f61871c37b13d
MD5 c3e199aad4ecfcc63882ebe9ffe233cd
BLAKE2b-256 f437ea0871a5ea9af976871e390133f52a21852530bd6e83fa054e50af169471

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.2.0.tar.gz:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file parquet_file_fs-0.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for parquet_file_fs-0.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 07e9e4665b30a263c66251584b76bf5f0730d1e0665cff85f302ca122a14d6bc
MD5 deba98bbee1aba8f56fbfd6a45c21bee
BLAKE2b-256 d6ce0d15b79950b4d21667b9e14d95ae14cac57e51854282ef7213f73da393bd

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.2.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file parquet_file_fs-0.2.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for parquet_file_fs-0.2.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 838c1e8dfd4e9c1ba5b5feb06fca8440188659ba28f90b20215915e1ff6ac293
MD5 e3c158520c83e4345ace296263f8f440
BLAKE2b-256 f31e3e1c947425cabca7939f3eb42e9d839e1c36695c5c8f125a3b0b251781c2

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.2.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.2.0 This release

3 files

0.1.0

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page