upathlib
Project description
upathlib
This package defines some APIs for working with a cloud blob store (aka "object store"). The motivation is to provide a "unified" or "universal" API for blob store clients, including local file systems.
The APIs follow the style of the standard library pathlib where appropriate.
Attention is focused on identifying the most essential functionalities
while working with a blob store for data processing.
Functionalities in a local file system that are non-essential in these tasks---such
such as symbolic links, fine-grained permissions,
and various access modes---are ignored.
Local (POSIX) file system is treated as one particular case,
and is implemented by LocalUpath.
Clients for Azure and GCP blob stores are implemented by subclasses
AzureBlobUpath and GcpBlobUpath, respectively. Users may want to
add very thin wrappers to handle credentials. Clients for other public-cloud blob stores will be added as needed.
One use case is the biglist package.
The class Biglist takes a Upath object to indicate its location of storage.
It does not care whether the storage is local, or in Azure or GCP or AWS S3 or other
cloud blob store---it simply uses the common API to operate the storage.
This package is published to pypi. To install, does
$ pip3 install upathlib
$ pip3 install upathlib[azure]
$ pip3 install upathlib[gcp]
$ pip3 install upathlib[azure,gcp]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
