Skip to main content

A HTTP and FTP parallel file downloader.

Project description

Latest PyPI version

A parallel file downloader using asyncio. parfive can handle downloading multiple files in parallel as well as downloading each file in a number of chunks.

Usage

asciicast demo of parfive

parfive works by creating a downloader object, appending files to it and then running the download. parfive has a synchronous API, but uses asyncio to paralellise downloading the files.

A simple example is:

from parfive import Downloader
dl = Downloader()
dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./")
files = dl.download()

Parfive also bundles a CLI. The following example will download the two files concurrently.:

$ parfive 'http://212.183.159.230/5MB.zip' 'http://212.183.159.230/10MB.zip'
$ parfive --help
usage: parfive [-h] [--max-conn MAX_CONN] [--overwrite] [--no-file-progress]
              [--directory DIRECTORY] [--print-filenames]
              URLS [URLS ...]

Parfive, the python asyncio based downloader

positional arguments:
  URLS                  URLs of files to be downloaded.

optional arguments:
  -h, --help            show this help message and exit
  --max-conn MAX_CONN   Number of maximum connections.
  --overwrite           Overwrite if the file exists.
  --no-file-progress    Show progress bar for each file.
  --directory DIRECTORY
                        Directory to which downloaded files are saved.
  --print-filenames     Print successfully downloaded files's names to stdout.

Results

parfive.Downloader.download returns a parfive.Results object, which is a list of the filenames that have been downloaded. It also tracks any files which failed to download.

Handling Errors

If files fail to download, the urls and the response from the server are stored in the Results object returned by parfive.Downloader. These can be used to inform users about the errors. (Note, the progress bar will finish in an incomplete state if a download fails, i.e. it will show 4/5 Files Downloaded).

The Results object is a list with an extra attribute errors, this property returns a list of named tuples, where these named tuples contains the .url and the .response, which is a aiohttp.ClientResponse or a aiohttp.ClientError object.

Installation

parfive is available on PyPI, you can install it with pip:

pip install parfive

or if you want to use FTP downloads:

pip install parfive[ftp]

Requirements

  • Python 3.7 or above

  • aiohttp

  • tqdm

  • aioftp (for downloads over FTP)

Licence

MIT Licensed

Authors

parfive was written by Stuart Mumford.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parfive-2.1.0.tar.gz (34.0 kB view details)

Uploaded Source

Built Distribution

parfive-2.1.0-py3-none-any.whl (27.6 kB view details)

Uploaded Python 3

File details

Details for the file parfive-2.1.0.tar.gz.

File metadata

  • Download URL: parfive-2.1.0.tar.gz
  • Upload date:
  • Size: 34.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for parfive-2.1.0.tar.gz
Algorithm Hash digest
SHA256 cd6cb419242130730cf41d4cf6f284ebf50f1a71ce6c9508e0467ec98f17dc8e
MD5 3eea9c15a18d9837d62320d013a7ab28
BLAKE2b-256 dc40b464df95057770f4e5110bf72c22d525009aad4d2c742baa7f2e67f554e9

See more details on using hashes here.

File details

Details for the file parfive-2.1.0-py3-none-any.whl.

File metadata

  • Download URL: parfive-2.1.0-py3-none-any.whl
  • Upload date:
  • Size: 27.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for parfive-2.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a935849ce7143f1ceda9b10c6fe68e082cdc10ddc5fe77c9c2c333132fb64dbe
MD5 0f1d2ac602f09934ff8080d550075e51
BLAKE2b-256 f5c1eedc928b34cd2f932eff4c6920da566fa06fb080d209ea7645be4e2327b1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page