Unified storage interface.
Project description
fw-storage
Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.
Supported storage backends:
fs://
- Local file-systems3://
- Amazon S3gs://
- Google Cloud Storageaz://
- Azure Blob Storage
Installation
Add as a poetry
dependency to your project:
poetry add fw-storage
The local file-system storage backend has no extra package dependencies. To use
any other backend, though, fw-storage
needs to be installed with extras. For
example, in order to access S3:
poetry add fw-storage --extras s3
Alternatively, install with all
extras to get everything:
poetry add fw-storage --extras all
Usage
from fw_storage import get_storage
# instantiate storage with URL
fs = get_storage("fs:///tmp")
# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))
# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2
# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7
# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"
# remove one or more objects
fs.rm("test", recurse=True)
Configuration
Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:
In addition, az://
can be configured with the envvar AZ_ACCESS_KEY
.
Development
Install the project using poetry
and enable pre-commit
:
poetry install -E all
pre-commit install
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for fw_storage-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9736fc8d705a2b4aac07f3f641b9061ac42e1432ff7d4ae820921b9d73fbc13 |
|
MD5 | 7bc57386fcffe6e53c9ca0a2a68cf4ac |
|
BLAKE2b-256 | a7b9255f53e3909583a52a7e82a82d6a99e1a0e24f4d687fec32390ed23f479f |