fw-storage
Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.
Supported storage backends:
fs://- Local file-systems3://- Amazon S3gs://- Google Cloud Storageaz://- Azure Blob Storage
Installation
Add as a poetry dependency to your project:
poetry add fw-storage
Usage
from fw_storage import create_storage_client
# instantiate storage with URL
fs = create_storage_client("fs:///tmp")
# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))
# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2
# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7
# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"
# remove one or more objects
fs.rm("test", recurse=True)
Configuration
Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:
| Storage | Config docs |
|---|---|
s3:// |
Boto |
gs:// |
|
az:// |
Azure |
In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.
Development
Install the project using poetry and enable pre-commit:
poetry install -E all
pre-commit install
License
1 maintainer
Metadata
1 maintainer
Release files for fw-storage 3.11.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fw_storage-3.11.3-py3-none-any.whl | Python 3 | none | any | Details |
Release files / fw_storage-3.11.3-py3-none-any.whl
| Download URL | fw_storage-3.11.3-py3-none-any.whl |
|---|---|
| Size | 46.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8cd0511c3e65b2c5637801877176dc7ddd5bad317215eb7db21c4398dc991bd5
|
|
BLAKE2b-256 checksum How to use checksums |
d4c96e026297643761d25320362097795a2e9cc61f8aa52cd7f90ba5a791c510
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.11 {"installer":{"name":"uv","version":"0.12.11","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Alpine Linux","version":"3.25.0_alpha20260805","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
