Azure storage transfer tool with AzCopy-like features
AzCopy-like OS independent Azure storage blob and file share transfer tool
blobxfer is on PyPI and can be installed via:
pip install blobxfer
blobxfer is compatible with Python 2.7 and 3.3+. To install for Python 3, some distributions may use pip3 instead. If you do not want to install blobxfer as a system-wide binary and modify system-wide python packages, use the --user flag with pip or pip3.
blobxfer is also on Docker Hub, and the Docker image for Linux can be pulled with the following command:
docker pull alfpark/blobxfer
Please see example usage below on how to use the docker image.
If you encounter difficulties installing the script, it may be due to the cryptography dependency. Please ensure that your system is able to install binary wheels provided by these dependencies (e.g., on Windows) or is able to compile the dependencies (i.e., ensure you have a C compiler, python, ssl, and ffi development libraries/headers installed prior to invoking pip). For instance, to install blobxfer on a fresh Ubuntu 14.04/16.04 installation for Python 2.7, issue the following commands:
apt-get update apt-get install -y build-essential libssl-dev libffi-dev libpython-dev python-dev python-pip pip install --upgrade blobxfer
If you need more fine-grained control on installing dependencies, continue reading this section. Depending upon the desired mode of authentication with Azure and options, the script will require the following packages, some of which will automatically pull required dependent packages. Below is a list of dependent packages:
You can install these packages using pip, easy_install or through standard setup.py procedures. These dependencies will be automatically installed if using a package-based install or setup.py. The required versions of these dependent packages can be found in setup.py.
The blobxfer.py script allows interacting with storage accounts using any of the following methods: (1) management certificate, (2) shared account key, (3) SAS key. The script can, in addition to working with single files, mirror entire directories into and out of containers or file shares from Azure Storage, respectively. File and block/page level MD5 integrity checking is supported along with various transfer optimizations, built-in retries, user-specified timeouts, and client-side encryption.
Program parameters and command-line options can be listed via the -h switch. Please invoke this first if you are unfamiliar with blobxfer operation as not all options are explained below. At the minimum, three positional arguments are required: storage account name, container or share name, and local resource. Additionally, one of the following authentication switches must be supplied: --subscriptionid with --managementcert, --storageaccountkey, or --saskey. Do not combine different authentication schemes together.
Environment variables BLOBXFER_STORAGEACCOUNTKEY, BLOBXFER_SASKEY, and BLOBXFER_RSAKEYPASSPHRASE can take the place of --storageaccountkey, --saskey, and --rsakeypassphrase respectively if you do not want to expose credentials on a command line.
It is generally recommended to use SAS keys wherever appropriate; only HTTPS transport is used in the script. Please note that when using SAS keys that only container- or fileshare-level SAS keys will allow for entire directory uploading or container/fileshare downloading. The container/fileshare must also have been created beforehand if using a service SAS, as containers/fileshares cannot be created using service SAS keys. Account-level SAS keys with a signed resource type of c or container will allow containers/fileshares to be created with SAS keys.
The following examples show how to invoke the script with commonly used options. Note that the authentication parameters are missing from the below examples. You will need to select a preferred method of authenticating with Azure and add the authentication switches (or as environment variables) as noted above.
The script will attempt to perform a smart transfer, by detecting if the local resource exists. For example:
blobxfer mystorageacct container0 mylocalfile.txt
Note: if you downloaded the script directly from github, then you should append .py to the blobxfer command.
If mylocalfile.txt exists locally, then the script will attempt to upload the file to container0 on mystorageacct. If the file does not exist, then it will attempt to download the resource. If the desired behavior is to download the file from Azure even if the local file exists, one can override the detection mechanism with --download. --upload is available to force the transfer to Azure storage. Note that specifying a particular direction does not force the actual operation to occur as that depends on other options specified such as skipping on MD5 matches. Note that you may use the --remoteresource flag to rename the local file as the blob name on Azure storage if uploading, however, --remoteresource has no effect if uploading a directory of files. Please refer to the --collate option as explained below.
If the local resource is a directory that exists, the script will attempt to mirror (recursively copy) the entire directory to Azure storage while maintaining subdirectories as virtual directories in Azure storage. You can disable the recursive copy (i.e., upload only the files in the directory) using the --no-recursive flag.
To upload a directory with files only matching a Unix-style shell wildcard pattern, an example commandline would be:
blobxfer mystorageacct container0 mylocaldir --upload --include '**/*.txt'
This would attempt to recursively upload the contents of mylocaldir to container0 for any file matching the wildcard pattern *.txt within all subdirectories. Include patterns can be applied for uploads as well as downloads. Note that you will need to prevent globbing by your shell such that wildcard expansion does not take place before script interprets the argument. If --include is not specified, all files will be uploaded or downloaded for the specific context.
To download an entire container from your storage account, an example commandline would be:
blobxfer mystorageacct container0 mylocaldir --remoteresource .
Assuming mylocaldir directory does not exist, the script will attempt to download all of the contents in container0 because “.” is set with --remoteresource flag. To download individual blobs, one would specify the blob name instead of “.” with the --remoteresource flag. If mylocaldir directory exists, the script will attempt to upload the directory instead of downloading it. If you want to force the download direction even if the directory exists, indicate that with the --download flag. When downloading an entire container, the script will attempt to pre-allocate file space and recreate the sub-directory structure as needed.
To collate files into specified virtual directories or local paths, use the --collate flag with the appropriate parameter. For example, the following commandline:
blobxfer mystorageacct container0 myvhds --upload --collate vhds --autovhd
If the directory myvhds had two vhd files a.vhd and subdir/b.vhd, these files would be uploaded into container0 under the virtual directory named vhds, and b.vhd would not contain the virtual directory subdir; thus, flattening the directory structure. The --autovhd flag would automatically enable page blob uploads for these files. If you wish to collate all files into the container directly, you would replace --collate vhds with --collate .
To strip leading components of a path on upload, use --strip-components with a number argument which will act similarly to tar’s --strip-components=NUMBER parameter. This parameter is only applied during an upload.
To encrypt or decrypt files, the option --rsapublickey and --rsaprivatekey is available. This option requires a file location for a PEM encoded RSA public or private key. An optional parameter, --rsakeypassphrase is available for passphrase protected RSA private keys.
To encrypt and upload, only the RSA public key is required although an RSA private key may be specified. To download and decrypt blobs which are encrypted, the RSA private key is required.
blobxfer mystorageacct container0 myblobs --upload --rsapublickey mypublickey.pem
The above example commandline would encrypt and upload files contained in myblobs using an RSA public key named mypublickey.pem. An RSA private key may be specified instead for uploading (public parts will be used).
blobxfer mystorageacct container0 myblobs --remoteresource . --download --rsaprivatekey myprivatekey.pem
The above example commandline would download and decrypt all blobs in the container container0 using an RSA private key named myprivatekey.pem. An RSA private key must be specified for downloading and decryption of encrypted blobs.
Currently only the FullBlob encryption mode is supported for the parameter --encmode. The FullBlob encryption mode either uploads or downloads Azure Storage .NET/Java compatible client-side encrypted block blobs.
Please read important points in the Encryption Notes below for more information.
To transfer to an Azure Files share, specify the --fileshare option and specify the share name as the second positional argument.
blobxfer mystorageacct myshare localfiles --fileshare --upload
The above example would upload all files in the localfiles directory to the share named myshare. Encryption/decryption options are compatible with Azure Files as the destination or source. Please refer to this MSDN article for features not supported by the Azure File Service.
An example execution for uploading the host path /example/host/path to a storage container named container0 would be:
docker run --rm -t -v /example/host/path:/path/in/container alfpark/blobxfer mystorageacct container0 /path/in/container --upload
Note that docker volume mount mappings must be crafted with care to ensure consistency with directory depth between the host and the container. Optionally, you can utilize the --strip-components flag to remove leading path components as desired.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|blobxfer-0.12.1-py2.py3-none-any.whl (41.8 kB) Copy SHA256 Checksum SHA256||py2.py3||Wheel||Dec 9, 2016|
|blobxfer-0.12.1.tar.gz (49.6 kB) Copy SHA256 Checksum SHA256||–||Source||Dec 9, 2016|