Skip to main content

Dask interface to Azure-Datalake Gen1 and Gen2 Storage

Warning: this code is experimental and untested.

Quickstart

This package is on PyPi and can be installed using:

pip install adlfs

In your code, call:

from fsspec.registry import known_implementations

To use the Gen1 filesystem:

known_implementations[‘adl’] = {‘class’: ‘adlfs.AzureDatalakeFileSystem’}

To use the Gen2 filesystem:

known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}

This allows operations such as: import dask.dataframe as dd storage_options={ ‘tenant_id’: TENANT_ID, ‘client_id’: CLIENT_ID, ‘client_secret’: CLIENT_SECRET, ‘storage_account’: STORAGE_ACCOUNT, ‘filesystem’: FILESYSTEM, } dd.read_csv(‘abfs://folder/file.csv’, storage_options=STORAGE_OPTIONS}

Details

The package includes pythonic filesystem implementations for both Azure Datalake Gen1 and Azure Datalake Gen2, that facilitate interactions with both Azure Datalake implementations with Dask, using the intake/filesystem_spec base class.

Operations against both Gen1 and Gen2 datalakes currently require an Azure ServicePrincipal with suitable credentials to perform operations on the resources of choice.

Operations on the Azure Gen1 Datalake are implemented by leveraging multiple inheritance from both the fsspec.AbstractFileSystem and the Azure Python Gen1 Filesystem library, while operations against the Azure Gen2 Datalake are implemented by using subclassing the fsspec.AbstractFileSystem and leveraging the Azure Datalake Gen2 API. Note that the Azure Datalake Gen2 API allows calls to using either the ‘http://’ or ‘https://’ protocols, designated by an ‘abfs[s]://’ protocol. Under the hood in adlfs, this will always happen using ‘https://’ using the requests library.

An Azure Datalake Gen2 url takes the following form, which is replicated in the adlfs library, for the sake of consistency: ‘abfs[s]://{storage_account}/{filesystem}/{folder}/{file}’

Currently, when using either the ‘adl://’ or ‘abfs://’ protocols in a dask operation, it is required to explicitly declare the storage_options, as described in the Dask documentation. The intent is to eliminate this requirement for (at at minimum) Gen2 operations, by having the adlfs library parse the filesystem name

Metadata

Release files for adlfs 0.0.9.post0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for adlfs 0.0.9.post0
File Size Uploaded Provenance Actions
adlfs-0.0.9.post0.tar.gz 7.4 kB Without provenance Details for adlfs-0.0.9.post0.tar.gz

Release files / adlfs-0.0.9.post0.tar.gz

Download URL adlfs-0.0.9.post0.tar.gz
Size 7.4 kB
Tags Source
SHA-256 checksum
How to use checksums
0109fa61800d13b95275e6708cc2754b1b7ec92f586f6b270e05b307c8e60030
BLAKE2b-256 checksum
How to use checksums
56e9b65fce98f120e5f9da638b35e8bd110403455cc75b0abe7b87f540db14f2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/2.7.15rc1

Security for adlfs 0.0.9.post0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.9.post0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page