parquet-file-fs

parquet-file-fs

ls, cat and glob the files stored inside parquet shards — a read-only fsspec filesystem with a Rust core.

CI python-ci PyPI License

Read-only fsspec filesystem over parquet "archive" shards — parquet files where each row is one file (a path column + a content column). Rust core (arrow/parquet + object_store) via maturin; built for handing datasets to coding agents: ls, cat, open, glob, walk over shard contents without loading whole shards into memory.

Install

pip install parquet-file-fs

Usage

from parquet_file_fs import ParquetFileSystem

fs = ParquetFileSystem("data/*.parquet")            # local glob
fs = ParquetFileSystem("s3://bucket/shard-*.parquet")  # S3 (env credentials)
fs = ParquetFileSystem(["a.parquet", "b.parquet"])  # multi-shard

fs.ls("images", detail=False)     # ['images/a.png', 'images/b.png']
fs.cat_file("labels/a.json")      # b'{"route": "agentic"}'
fs.glob("**/*.json")              # index-only, no content decoding
fs.info("images/a.png")           # {'name': ..., 'size': 5, 'type': 'file',
                                  #  'metadata': {<extra parquet columns>}}
with fs.open("readme.txt", "r") as f:
    print(f.read())

Column names are auto-detected (path/filename/file_name/key and content/data/bytes) or set explicitly:

fs = ParquetFileSystem("odd.parquet", path_column="file_name",
                       content_column="image_bytes")

Duplicate paths across shards raise by default; pass on_duplicate="first" or "last" to pick a shard instead.

Protocol adapters

Built-in: local paths / file://, s3:// (credentials from the environment), http://, https://. Any other scheme can be registered — either an fsspec filesystem or a tiny duck-typed object:

import gcsfs
from parquet_file_fs import ParquetFileSystem, register_adapter

register_adapter("gs", gcsfs.GCSFileSystem())
fs = ParquetFileSystem("gs://bucket/data/*.parquet")

Custom adapter interface: size(url) -> int, read_range(url, offset, length) -> bytes (must return exactly length bytes; callers never request past EOF), glob(pattern) -> list[str].

Glob patterns are supported for local paths and s3://; pass concrete URLs for http(s)://.

Notes

  • Read-only: mutation methods raise NotImplementedError.
  • The index is built from parquet footers + the path column only; file content decodes lazily per row group (small LRU cache). Exact file sizes are computed on first info/ls(detail=True) per row group.
  • Registered with fsspec as protocol pfs.

Development

uv sync                 # dev deps from [dependency-groups]
uv run maturin develop  # build + install the extension
uv run pytest           # Python tests
cargo test              # Rust tests
cargo clippy --all-targets -- -D warnings
cargo fmt --all

Without uv:

python3 -m venv .venv && .venv/bin/pip install maturin pytest pyarrow pyyaml fsspec
.venv/bin/maturin develop && .venv/bin/pytest

Releasing

Commits on master drive release-please: it opens a release PR that bumps pyproject.toml, Cargo.toml (via the x-release-please-version marker) and Cargo.lock. Merging that PR tags the release and publishes wheels (linux x86_64, macOS arm64) plus an sdist to PyPI through Trusted Publishing — no API tokens in the repo. A failed publish can be re-run with workflow_dispatch on the release-please workflow — that path also publishes the current master version directly, which is the way out if the release PR itself can't be opened.

Opening that PR requires Allow GitHub Actions to create and approve pull requests, enabled at both the repo (Settings → Actions → General) and the account level (github.com/settings/actions). Where that policy is unavailable, add a PAT with contents: write + pull-requests: write as the RELEASE_PLEASE_TOKEN secret and the workflow uses it instead.

License

MIT OR Apache-2.0, at your option.

Project links

Data verified by PyPI on Jul 24, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ransomware from gravatar.com ransomware

Credits

Author: 4thel00z

GitHub Statistics

Data verified by PyPI on Jul 24, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT OR Apache-2.0
View SPDX License List

Requires

Python >=3.9

Tags

parquet fsspec filesystem virtual-filesystem arrow columnar dataset s3 pyo3 rust

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parquet_file_fs-0.1.0.tar.gz (75.7 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

parquet_file_fs-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.2 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

parquet_file_fs-0.1.0-cp39-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

File details

Details for the file parquet_file_fs-0.1.0.tar.gz.

File metadata

  • Download URL: parquet_file_fs-0.1.0.tar.gz
  • Upload date:
  • Size: 75.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for parquet_file_fs-0.1.0.tar.gz
Algorithm Hash digest
SHA256 9fb4675509346e631a3b4200b82a988a5ceb83486c34c71d4f9b3f64b6342be7
MD5 c94d18a008968edc084207a5ffb39669
BLAKE2b-256 2ca7a4ed4b6aa55dbcf1d404f45d1de3311a08232bc1904edaa7dabc82f5bf46

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.1.0.tar.gz:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file parquet_file_fs-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for parquet_file_fs-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0ca5f877025739e1170fbf608ab33778556c1c9cd0472925c835d9849e8916f5
MD5 7ecf739983da3292a309d5a2ee4e03f0
BLAKE2b-256 c6282e99d6055b527dfee1a871c66738e91bc15ce504226a035aae895272efd3

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.1.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file parquet_file_fs-0.1.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for parquet_file_fs-0.1.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 26247371c2a1bc798d44d2f8416a253b8737f47003e379226aa4ebec2db9bac6
MD5 98eeb77905a3eebdf24784e56434c93f
BLAKE2b-256 90048f0092f041849e9947ca1ce2cafce12c52052c0bace6c1c2d3b84c18c465

See more details on using hashes here.

Provenance

The following attestation bundles were made for parquet_file_fs-0.1.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release-please.yaml on 4thel00z/parquet-file-fs

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.0

3 files

This release

0.1.0 This release

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page