fastdownload

Easily download, verify, and extract archives

If you have datasets or other archives that you want to make available to your users, and ensure they always have the latest versions and that they are downloaded correctly, fastdownload can help.

Install

Using pip:

pip install fastdownload

...or using conda:

conda install -c fastai fastdownload

What's this about?

The situation where you might want to use fastdownload is where you have one or more URLs pointing at some archives you want to make available, and you want to ensure that your users download those archives correctly, have the latest version, and that it's as easy as possible for them to access the information in those archives.

Your user just calls a single method, FastDownload.get, passing the URL required, and the URL will be downloaded and extracted to the directories you choose. The path to the extracted file is returned. If that URL has already been downloaded, then the cached archive or contents will be used automatically. However, if that size or hash of the archive is different to what it should be, then the user will be informed, and a new version will be downloaded.

In the future, you may want to update one or more of your archives. When you do so, fastdownload will ensure your users have the latest version, by checking their downloaded archives against your updated file size and hash information.

For instance, fastai uses fastdownload to provide access to datasets for deep learning. fastai users can download and extract them with a single command, using the return value to access the files. The files are automatically placed in appropriate subdirectories of a .fastai folder in the user's homedir. If a dataset is updated, users are informed the next time they use the dataset, and the latest version is automatically downloaded and extracted for them.

Usage: downloading files

When your users download an archive, fastdownload will automatically save it to a directory, check if the size and hash matches, and extract the contents. Minimal usage for downloading and extracting is:

from fastdownload import FastDownload
d = FastDownload()
path = d.get('https://...')

After this, path will contain the path where the extracted files are located. By default, archives are saved to {base}/archive, and extracted to {base}/data. {base} defaults to ~/.fastdownload. If there is more than one file or folder in the root of the downloaded archive, then a new folder is created in data for the contents.

Instead of get, use download to download the URL without extracting it, or extract to extract the URL without downloading it (assuming it's already been downloaded to the archive directory). All of these methods accept a force parameter which will download/extract the archive even if it's already present.

You can change any or all of the base, archive, and data paths by passing them to FastDownload:

d = FastDownload(base='~/.mypath', archive='downloaded', data='extracted')

You can remove the cached archive file and/or the extracted contents with rm:

d.rm('https://...')

Usage: making archives available to download

fastdownload will add a file download_checks.py to your Python module which contains file sizes and hashes for your archives. The file is located in the same directory as a module you choose, e.g.:

d = FastDownload(module=fastai.some_module)

Then use update to create or update the size and hash for a URL:

d.update('https://...')

You will now find there is a file called download_checks.py in the same directory where fastai.some_module is located, which contains a Python dict with the URL, size, and hash for this file. If you've downloaded this file before to your archive path then it will be used, instead of downloading a new copy. Use get(force=True) first to download a new copy if even you have it in your archive.

Config file

If there is a file called config.ini in your base directory, then keys archive and data will be used as the default values for FastDownload. The file should be in configparser format. Here's a sample config.ini:

[DEFAULT]         
archive = downloaded
data = extracted

If there is no ini file present, one will be automatically created for for you using the details you pass to FastDownload.

You can add any additional key/value pairs to the config file that you want. When you call FastDownload.get pass extract_key to use a key other than data for choosing a location to extract to.