Skip to main content

Azure blob storage filesystem for PyFilesystem2

Project description

PyPI made-with-python Code style: black Tests GitHub contributors GitHub commit activity GitHub last commit (branch) GitHub pull requests License: MIT Code of Conduct

fs-azureblob

Installation

The package can be installed via pip:

pip install git+https://github.com/Breakthrough-Energy/fs-azureblob

Or by cloning the repository and installing directly:

git clone https://github.com/Breakthrough-Energy/fs-azureblob
cd fs-azureblob
pip install .

Either approach will also install the core fs package if it's not already installed.

Usage

This library implements the pyfilesystem API for blob storage containers in a general purpose storage account. There are implementations for the original blob storage, which uses a flat namespace with virtual directories, and accounts with hierarchical namespace enabled, which adds native directory support as well as other features. The type of account must be specified when a filesystem is instantiated: use the azblob protocol, or BlobFS class for accounts with a flat namespace, or the azblobv2 protocol or the BlobFSV2 class for accounts with a hierarchical namespace.

Opener

Use fs.open_fs to open a filesystem with an azure blob FS URL, where protocol is either azblob or azblobv2:

import fs
my_fs = fs.open_fs("[protocol]://[account_name]:[account_key]@[container]")

Constructor

The BlobFS (or BlobFSV2) class can also be instantiated directly

from fs.azblob import BlobFS
my_fs = BlobFS(account_name, container, account_key)

using the following arguments:

  • account_name: the name of the storage account
  • container: the blob container
  • account_key: optional, but required for write operations or depending on the storage account access policies

Resource Info

Users can call getinfo for the basic and details namespaces, however support for setinfo is limited, as these properties are enforced by azure (e.g. last modified time). There is a custom namespace called blob which can be used to set metadata on a blob, in the form of key value pairs which must be valid http headers.

Additionally, the v2 filesystem for hierarchical namespaces supports posix permissions, so the access namespaces is supported for getinfo calls, which includes this information.

See docs for more details.

Note

The following can be ignored if using an account with hierarchical namespace.

Since blob storage uses a flat namespace (directories don't really exist), we create a placeholder file to represent them, always named .fs_azblob. This is an empty blob which is created for new directories, removed when a directory is removed, and omitted from listdir results, so should be transparent to users. To use this package on a new blob storage container, nothing needs to be done. For usage on an existing container, one should create this structure using the azure portal, sdk, or preferred tool, to ensure this package will function as expected.

Additionally, this package is intended to operate on "block blobs". Other blob types include page blobs and append blobs. The package has not been tested on these types.

See also

  • fs, the core PyFilesystem2 library

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fs-azureblob-0.2.1.tar.gz (10.8 kB view details)

Uploaded Source

Built Distribution

fs_azureblob-0.2.1-py3-none-any.whl (11.9 kB view details)

Uploaded Python 3

File details

Details for the file fs-azureblob-0.2.1.tar.gz.

File metadata

  • Download URL: fs-azureblob-0.2.1.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for fs-azureblob-0.2.1.tar.gz
Algorithm Hash digest
SHA256 12536fc7ea5ac2a5b831d87b997475296e893006746ee2a653c56ee8ed22b0be
MD5 5ac8d9486602fa6ca54d0f9fe8fd25ff
BLAKE2b-256 1f24025def34a362ec5b827483d099facae7256a9a7dffde2bf9440ad01fcaa0

See more details on using hashes here.

File details

Details for the file fs_azureblob-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: fs_azureblob-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 11.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for fs_azureblob-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 55245b1ff9b5fe448aec13dfa600552d5773f9a6504dcaba80d2618299659c1e
MD5 b53a0fb717c1037420dd624dbd890e86
BLAKE2b-256 896c82e5e4afcce8a339e177662276b9e19701e07d6f140e0c5228a7e48e6df0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page