Azure blob storage filesystem for PyFilesystem2
Project description
fs-azureblob
Installation
The package can be installed via pip:
pip install git+https://github.com/Breakthrough-Energy/fs-azureblob
Or by cloning the repository and installing directly:
git clone https://github.com/Breakthrough-Energy/fs-azureblob
cd fs-azureblob
pip install .
Either approach will also install the core fs
package if it's not already installed.
Usage
This library implements the pyfilesystem API for blob storage containers in a general
purpose storage account. There are implementations for the original blob storage, which
uses a flat namespace with virtual directories, and accounts with hierarchical namespace
enabled, which adds native directory support as well as other features. The type of
account must be specified when a filesystem is instantiated: use the azblob
protocol,
or BlobFS
class for accounts with a flat namespace, or the azblobv2
protocol or the
BlobFSV2
class for accounts with a hierarchical namespace.
Opener
Use fs.open_fs
to open a filesystem with an azure blob
FS URL, where protocol
is
either azblob
or azblobv2
:
import fs
my_fs = fs.open_fs("[protocol]://[account_name]:[account_key]@[container]")
Constructor
The BlobFS
(or BlobFSV2
) class can also be instantiated directly
from fs.azblob import BlobFS
my_fs = BlobFS(account_name, container, account_key)
using the following arguments:
account_name
: the name of the storage accountcontainer
: the blob containeraccount_key
: optional, but required for write operations or depending on the storage account access policies
Resource Info
Users can call getinfo
for the basic
and details
namespaces, however support for
setinfo
is limited, as these properties are enforced by azure (e.g. last modified
time). There is a custom namespace called blob
which can be used to set metadata on a
blob, in the form of key value pairs which must be valid http headers.
Additionally, the v2 filesystem for hierarchical namespaces supports posix permissions,
so the access
namespaces is supported for getinfo
calls, which includes this
information.
See docs for more details.
Note
The following can be ignored if using an account with hierarchical namespace.
Since blob storage uses a flat namespace (directories don't really exist), we create a
placeholder file to represent them, always named .fs_azblob
. This is an empty blob
which is created for new directories, removed when a directory is removed, and omitted
from listdir
results, so should be transparent to users. To use this package on a new
blob storage container, nothing needs to be done. For usage on an existing container,
one should create this structure using the azure portal, sdk, or preferred tool, to
ensure this package will function as expected.
Additionally, this package is intended to operate on "block blobs". Other blob types include page blobs and append blobs. The package has not been tested on these types.
See also
- fs, the core PyFilesystem2 library
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file fs-azureblob-0.2.1.tar.gz
.
File metadata
- Download URL: fs-azureblob-0.2.1.tar.gz
- Upload date:
- Size: 10.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12536fc7ea5ac2a5b831d87b997475296e893006746ee2a653c56ee8ed22b0be |
|
MD5 | 5ac8d9486602fa6ca54d0f9fe8fd25ff |
|
BLAKE2b-256 | 1f24025def34a362ec5b827483d099facae7256a9a7dffde2bf9440ad01fcaa0 |
File details
Details for the file fs_azureblob-0.2.1-py3-none-any.whl
.
File metadata
- Download URL: fs_azureblob-0.2.1-py3-none-any.whl
- Upload date:
- Size: 11.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55245b1ff9b5fe448aec13dfa600552d5773f9a6504dcaba80d2618299659c1e |
|
MD5 | b53a0fb717c1037420dd624dbd890e86 |
|
BLAKE2b-256 | 896c82e5e4afcce8a339e177662276b9e19701e07d6f140e0c5228a7e48e6df0 |