Skip to main content

Add Azure dataset support to dtool

Project description

PyPi package

Features

  • Copy datasets to and from Azure storage

  • List all the datasets in a Azure storage account

  • Create datasets directly in Azure storage

Installation

To install the dtool-azure package:

pip install dtool-azure

Configuration

Install the Azure command line client via:

pip install azure-cli

(you may wish to install this in a virtual environment)

Then:

az login

To log into Azure.

Then you need to run (changing the resource name/group as appropriate):

az storage account show-connection-string --name jicinformatics --resource-group jic_informatics_resources_ukwest

Then create the file .config/dtool/dtool.json and add the Azure account name and key using the format below:

{
    "DTOOL_AZURE_ACCOUNT_KEY_<ACCOUNT NAME>": "<KEY HERE>"
}

Changing the account name and key as appropriate. For example if the account name was “jicinformatics” and the key “some-secret-token”:

{
    "DTOOL_AZURE_ACCOUNT_KEY_jicinformatics": "some-secret-token"
}

Usage

To copy a dataset from local disk (my-dataset) to an Azure storage account (jicinformatics) one can use the command below:

dtool copy ./my-dataset azure://jicinformatics/

To list all the datasets in an Azure storage account one can use the command below:

dtool ls azure://jicinformatics/

See the dtool documentation for more detail.

Configuring the local dtool Azure cache

When fetching items from a dataset, for example using the dtool item fetch command, the content of the item is cached in a file on local disk. The default cache directory is ~/.cache/dtool/azure.

One may want to change this directory. For example, if working on a HPC cluster to set it to a directory that lives on fast solid state disk. This can be achieved by setting the DTOOL_AZURE_CACHE_DIRECTORY environment variable. For example:

mkdir -p /tmp/dtool/azure
export DTOOL_AZURE_CACHE_DIRECTORY=/tmp/dtool/azure

Alternatively, when using the dtool command line interface one can add the DTOOL_AZURE_CACHE_DIRECTORY key to the ~/.config/dtool/dtool.json file. For example:

{
   "DTOOL_AZURE_CACHE_DIRECTORY": "/tmp/dtool/azure"
}

If the file does not exist one may need to create it.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dtool-azure-0.3.1.tar.gz (7.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page