Skip to main content

Unified storage interface.

Project description

fw-storage

Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.

Supported storage backends:

  • fs:// - Local file-system
  • s3:// - Amazon S3
  • gs:// - Google Cloud Storage
  • az:// - Azure Blob Storage

Installation

Add as a poetry dependency to your project:

poetry add fw-storage

Usage

from fw_storage import create_storage_client

# instantiate storage with URL
fs = create_storage_client("fs:///tmp")

# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))

# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2

# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7

# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"

# remove one or more objects
fs.rm("test", recurse=True)

Configuration

Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:

Storage Config docs
s3:// https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
gs:// https://google-auth.readthedocs.io/en/latest/reference/google.auth.html
az:// https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential

In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.

Development

Install the project using poetry and enable pre-commit:

poetry install -E all
pre-commit install

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

fw_storage-3.7.7-py3-none-any.whl (35.3 kB view details)

Uploaded Python 3

File details

Details for the file fw_storage-3.7.7-py3-none-any.whl.

File metadata

  • Download URL: fw_storage-3.7.7-py3-none-any.whl
  • Upload date:
  • Size: 35.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.1.3 CPython/3.13.3 Linux/5.15.154+

File hashes

Hashes for fw_storage-3.7.7-py3-none-any.whl
Algorithm Hash digest
SHA256 12a8cea022229af41219ee3bc4f22c154c0b7c78414c5cb5feed3f9d4ddf8b0c
MD5 94d74fc309baf3a15df0a820fb1258df
BLAKE2b-256 4e40a176fc2d2cdfef1f5f8021886b69a4997a76bf3f87f5c593cb4ed126a551

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page