Skip to main content

Unified storage interface.

Project description


Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.

Supported storage backends:

  • fs:// - Local file-system
  • s3:// - Amazon S3
  • gs:// - Google Cloud Storage
  • az:// - Azure Blob Storage


Add as a poetry dependency to your project:

poetry add fw-storage

The local file-system storage backend has no extra package dependencies. To use any other backend, though, fw-storage needs to be installed with extras. For example, in order to access S3:

poetry add fw-storage --extras s3

Alternatively, install with all extras to get everything:

poetry add fw-storage --extras all


from fw_storage import create_storage_client

# instantiate storage with URL
fs = create_storage_client("fs:///tmp")

# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))

# list objects, filtering with expressions
files = list("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2

# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7

# read object contents
file = fs.get("test/file1.dat") == b"content"

# remove one or more objects
fs.rm("test", recurse=True)


Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:

Storage Config docs

In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.


Install the project using poetry and enable pre-commit:

poetry install -E all
pre-commit install



Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

fw_storage-3.2.2-py3-none-any.whl (31.4 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page