A general purpose data downloading library.
Project description
fastdownload
Easily download, verify, and extract archives
If you have datasets or other archives that you want to make available to your users, and ensure they always have the latest versions and that they are downloaded correctly, fastdownload
can help.
Install
Using pip:
pip install fastdownload
...or using conda:
conda install -c fastai fastdownload
What's this about?
The situation where you might want to use fastdownload
is where you have one or more URLs pointing at some archives you want to make available, and you want to ensure that your users download those archives correctly, have the latest version, and that it's as easy as possible for them to access the information in those archives.
Your user just calls a single method, FastDownload.get
, passing the URL required, and the URL will be downloaded and extracted to the directories you choose. The path to the extracted file is returned. If that URL has already been downloaded, then the cached archive or contents will be used automatically. However, if that size or hash of the archive is different to what it should be, then the user will be informed, and a new version will be downloaded.
In the future, you may want to update one or more of your archives. When you do so, fastdownload
will ensure your users have the latest version, by checking their downloaded archives against your updated file size and hash information.
For instance, fastai
uses fastdownload
to provide access to datasets for deep learning. fastai
users can download and extract them with a single command, using the return value to access the files. The files are automatically placed in appropriate subdirectories of a .fastai
folder in the user's homedir. If a dataset is updated, users are informed the next time they use the dataset, and the latest version is automatically downloaded and extracted for them.
Usage: downloading files
When your users download an archive, fastdownload
will automatically save it to a directory, check if the size and hash matches, and extract the contents. Minimal usage for downloading and extracting is:
from fastdownload import FastDownload
d = FastDownload()
path = d.get('https://...')
After this, path
will contain the path where the extracted files are located. By default, archives are saved to {base}/archive
, and extracted to {base}/data
. {base}
defaults to ~/.fastdownload
. If there is more than one file or folder in the root of the downloaded archive, then a new folder is created in data
for the contents.
Instead of get
, use download
to download the URL without extracting it, or extract
to extract the URL without downloading it (assuming it's already been downloaded to the archive
directory). All of these methods accept a force
parameter which will download/extract the archive even if it's already present.
You can change any or all of the base
, archive
, and data
paths by passing them to FastDownload
:
d = FastDownload(base='~/.mypath', archive='downloaded', data='extracted')
You can remove the cached archive file and/or the extracted contents with rm
:
d.rm('https://...')
Usage: making archives available to download
fastdownload
will add a file download_checks.py
to your Python module which contains file sizes and hashes for your archives. The file is located in the same directory as a module you choose, e.g.:
d = FastDownload(module=fastai.some_module)
Then use update
to create or update the size and hash for a URL:
d.update('https://...')
You will now find there is a file called download_checks.py
in the same directory where fastai.some_module
is located, which contains a Python dict with the URL, size, and hash for this file. If you've downloaded this file before to your archive
path then it will be used, instead of downloading a new copy. Use get(force=True)
first to download a new copy if even you have it in your archive.
Config file
If there is a file called config.ini
in your base
directory, then keys archive
and data
will be used as the default values for FastDownload
. The file should be in configparser format. Here's a sample config.ini
:
[DEFAULT]
archive = downloaded
data = extracted
If there is no ini file present, one will be automatically created for for you using the details you pass to FastDownload
.
You can add any additional key/value pairs to the config file that you want. When you call FastDownload.get
pass extract_key
to use a key other than data
for choosing a location to extract to.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for fastdownload-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b791fa3406a2da003ba64615f03c60e2ea041c3c555796450b9a9a601bc0bbac |
|
MD5 | ea2955724c778270aecddc1b2be4906a |
|
BLAKE2b-256 | 4760ed35253a05a70b63e4f52df1daa39a6a464a3e22b0bd060b77f63e2e2b6a |