rfsspec
Rust python FSs
Implements s3, gcs, azure blob and HTTP backends for fsspec using Rust.
Respects concurrency of many simultaneous requests as made by fsspec, but
- does not need python asyncio,
- releases the GIL,
- can safely be called from multiple threads
- is probably NOT fork-safe
Limitations
Currently only the methods cat_ranges, cat and cat_file are supported, enough
to open a (consolidated) zarr dataset for reading.
Implementations
The http backend supports:
- headers
- method
- ranges
The s3 implementation supports configuration by environment variables and .aws files, and options
- profile
- endpoint_url
- anon
- region
- ranges
- requester-pays
The GCS backend supports:
- anon
- ranges
- requester-pays
- assumes credentials via env variables and gcloud CLI
The Azure blob backend supports
- anon
- ranges
- account/key auth (account always required)
Installation
> pip install -i https://pypi.anaconda.org/mdurant/simple rfsspec
Benchmarks
Latest results from examples/script.py:
Rust http: 0.1335
Rust s3: 0.1163
Rust anon s3: 0.0921
fsspec http: 0.1611
fsspec s3: 0.4758
fsspec anon s3: 0.4208
(multi-threaded benchmarks should be more meaningful)
Metadata
Release files for rfsspec 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| rfsspec-0.1.1-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.whl | CPython 3.11 | CPython 3.11 | Linux glibc 2.12+ x86-64 | Details for rfsspec-0.1.1-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
| rfsspec-0.1.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl | CPython 3.10 | CPython 3.10 | Linux glibc 2.12+ x86-64 | Details for rfsspec-0.1.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
| rfsspec-0.1.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl | CPython 3.9 | CPython 3.9 | Linux glibc 2.12+ x86-64 | Details for rfsspec-0.1.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
| rfsspec-0.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl | CPython 3.8 | CPython 3.8 | Linux glibc 2.12+ x86-64 | Details for rfsspec-0.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
Total release size: 25.3 MB
Release files / rfsspec-0.1.1-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
| Download URL | rfsspec-0.1.1-cp311-cp311-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
|---|---|
| Size | 6.3 MB |
| Tags | CPython 3.11 Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
6cf19d8286b755a82bdab13f166057915915f6a9735403c3b5fbaabb97236298
|
|
BLAKE2b-256 checksum How to use checksums |
12b5b74e7d06ae88c109cf6477a6798b269df0b8a52a57a5a00768dd5e1fe849
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Release files / rfsspec-0.1.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
| Download URL | rfsspec-0.1.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
|---|---|
| Size | 6.3 MB |
| Tags | CPython 3.10 Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
6b77f672e1c9fad8a812a366dd3e531f4b971f725d24d81a48e71db64a4bc262
|
|
BLAKE2b-256 checksum How to use checksums |
74a27cc1c928abff30f5cbd022d5e5469da3650ae361428c0fdd5af372b964be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Release files / rfsspec-0.1.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
| Download URL | rfsspec-0.1.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
|---|---|
| Size | 6.3 MB |
| Tags | CPython 3.9 Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
7ea03debe8b598b90d5430d32de593980f46c48c839e8578c710beffe09b2889
|
|
BLAKE2b-256 checksum How to use checksums |
45995b1c17714267fdcebc97ee4e735e6d34c6bcb2c01ebde2e709e6ba1f7526
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Release files / rfsspec-0.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
| Download URL | rfsspec-0.1.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl |
|---|---|
| Size | 6.3 MB |
| Tags | CPython 3.8 Linux glibc 2.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
9ff798cb1f3eba71e2d33009634b2a65b309ca03adb06b68b237adff800c0119
|
|
BLAKE2b-256 checksum How to use checksums |
bfad15a15c9e18af6e42202101e2a40d5599e898acb2e170f1ce1b75f7338f60
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Security for rfsspec 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.