fw-storage
Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.
Supported storage backends:
fs://- Local file-systems3://- Amazon S3gs://- Google Cloud Storageaz://- Azure Blob Storage
Installation
Add as a poetry dependency to your project:
poetry add fw-storage
Usage
from fw_storage import create_storage_client
# instantiate storage with URL
fs = create_storage_client("fs:///tmp")
# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))
# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2
# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7
# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"
# remove one or more objects
fs.rm("test", recurse=True)
Configuration
Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:
| Storage | Config docs |
|---|---|
s3:// |
Boto |
gs:// |
|
az:// |
Azure |
In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.
Development
Install the project using poetry and enable pre-commit:
poetry install -E all
pre-commit install
License
1 maintainer
Metadata
1 maintainer
Release files for fw-storage 3.11.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fw_storage-3.11.2-py3-none-any.whl | Python 3 | none | any | Details |
Release files / fw_storage-3.11.2-py3-none-any.whl
| Download URL | fw_storage-3.11.2-py3-none-any.whl |
|---|---|
| Size | 46.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1f42876bdc68294215c634fc39cc1b81b83c7b6910a9f75bbee83ed77bfd7f70
|
|
BLAKE2b-256 checksum How to use checksums |
53b10a4aa2d872b3da310f39fb2bd0f87d13e3df870470c724d651bebbc0ad62
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Alpine Linux","version":"3.25.0_alpha20260805","id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
