fw-storage
Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.
Supported storage backends:
fs://- Local file-systems3://- Amazon S3gs://- Google Cloud Storageaz://- Azure Blob Storage
Installation
Add as a poetry dependency to your project:
poetry add fw-storage
Usage
from fw_storage import create_storage_client
# instantiate storage with URL
fs = create_storage_client("fs:///tmp")
# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))
# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2
# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7
# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"
# remove one or more objects
fs.rm("test", recurse=True)
Configuration
Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:
| Storage | Config docs |
|---|---|
s3:// |
Boto |
gs:// |
|
az:// |
Azure |
In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.
Development
Install the project using poetry and enable pre-commit:
poetry install -E all
pre-commit install
License
1 maintainer
Metadata
1 maintainer
Release files for fw-storage 3.12.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fw_storage-3.12.0-py3-none-any.whl | Python 3 | none | any | Details |
Release files / fw_storage-3.12.0-py3-none-any.whl
| Download URL | fw_storage-3.12.0-py3-none-any.whl |
|---|---|
| Size | 46.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b90f108f323d8eacd10db357366782eafd4cde231b52acb5025cdd02e0958c7c
|
|
BLAKE2b-256 checksum How to use checksums |
92bb9ef3d46f011d2eef4ee9e177d7332278306354202c2b88aed22cd9774c75
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.3 CPython/3.13.15 Linux/5.15.154+
|
