azurebatchload 0.0.5b0

Download files in batches from Azure Blob Storage Containers

Azure Batch Load

High level Python wrapper around the Azure CLI to download or upload files in batches from or to Azure Blob Storage Containers. This project aims to be the missing functionality in the Python SDK of Azure Storage since there is no possibility to download or upload batches of files from or to containers. The only option in the Azure Storage Python SDK is downloading file by file, which takes a lot of time.

Note: Azure CLI has to be installed and configured.

Check if Azure CLI is installed through terminal:

az --version

---

Usage example Download:

1. Using the standard environment variables

Azure-batch-load automatically checks for environment variables: AZURE_STORAGE_CONNECTION_STRING, AZURE_STORAGE_KEYand AZURE_STORAGE_ACCOUNT. So if the connection_string or storage_key + storage_account are set as environment variables, we can leave the argument connection_string, account_key and account_name empty:

import os
from azurebatchload import DownloadBatch

if __name__ == '__main__':
    az_batch = DownloadBatch(
        destination='../pdfs',
        source='blobcontainername',
        pattern='*.pdf'
    )
    az_batch.download()

2. Using own environment variables

If we use other names for the environment variables, we can define the arguments connection_string, account_key and account_name in our function:

import os
from azurebatchload import DownloadBatch

if __name__ == '__main__':
    az_batch = DownloadBatch(
        destination='../pdfs',
        source='blobcontainername',
        connection_string=os.environ.get("connection_string"),
        pattern='*.pdf'
    )
    az_batch.download()

Or with key and name:

import os
from azurebatchload import DownloadBatch

if __name__ == '__main__':
    az_batch = DownloadBatch(
        destination='../pdfs',
        source='blobcontainername',
        account_key=os.environ.get("account_key"),
        account_name=os.environ.get("account_name"),
        pattern='*.pdf'
    )
    az_batch.download()

---

Usage example upload:

1. Using the standard environment variables

import os
from azurebatchload import UploadBatch

if __name__ == '__main__':
    az_batch = UploadBatch(
        destination='blobcontainername',
        source='../pdf',
        pattern='*.pdf'
    )
    az_batch.upload()

---

For more information about file pattern matching in the pattern argument, see Python Documentation.