A parallel file downloader using asyncio.
Usage
parfive works by creating a downloader object, appending files to it and then running the download. parfive has a synchronous API, but uses asyncio to paralellise downloading the files.
A simple example is:
from parfive import Downloader
dl = Downloader()
dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./")
files = dl.download()
Results
parfive.Downloader.download returns a parfive.Results object, which is a list of the filenames that have been downloaded. It also tracks any files which failed to download.
Handling Errors
If files fail to download, the urls and the response from the server are stored in the Results object returned by parfive.Downloader. These can be used to inform users about the errors. (Note, the progress bar will finish in an incomplete state if a download fails, i.e. it will show 4/5 Files Downloaded).
The Results object is a list with an extra attribute errors, this property returns a list of named tuples, where these named tuples contains the .url and the .response, which is a aiohttp.ClientResponse or a aiohttp.ClientError object.
Installation
parfive is available on PyPI, you can install it with pip:
pip install parfive
or if you want to use FTP downloads:
pip install parfive[ftp]
Requirements
Python 3.5+
aiohttp
tqdm
aioftp (for downloads over FTP)
Licence
MIT Licensed
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file parfive-0.2.3.tar.gz.
File metadata
- Download URL: parfive-0.2.3.tar.gz
- Upload date:
- Size: 14.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5936cd5897c5b26b2e6990f10f01af8fec891e61d26b4d28fcca2db7e4cedc5c
|
|
| MD5 |
3854256a727a636dd2dc4b37442ca3f3
|
|
| BLAKE2b-256 |
1d13092a4a45ff64c2cedf8f70a738b52163740dee3e23d17b132b09d447cf6f
|
