Skip to main content
Latest PyPI version

A parallel file downloader using asyncio. parfive can handle downloading multiple files in parallel as well as downloading each file in a number of chunks.

Usage

asciicast demo of parfive

parfive works by creating a downloader object, appending files to it and then running the download. parfive has a synchronous API, but uses asyncio to paralellise downloading the files.

A simple example is:

from parfive import Downloader
dl = Downloader()
dl.enqueue_file("http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt", path="./")
files = dl.download()

Parfive also bundles a CLI. The following example will download the two files concurrently.:

$ parfive 'http://212.183.159.230/5MB.zip' 'http://212.183.159.230/10MB.zip'
$ parfive --help
usage: parfive [-h] [--max-conn MAX_CONN] [--overwrite] [--no-file-progress]
              [--directory DIRECTORY] [--print-filenames]
              URLS [URLS ...]

Parfive, the python asyncio based downloader

positional arguments:
  URLS                  URLs of files to be downloaded.

optional arguments:
  -h, --help            show this help message and exit
  --max-conn MAX_CONN   Number of maximum connections.
  --overwrite           Overwrite if the file exists.
  --no-file-progress    Show progress bar for each file.
  --directory DIRECTORY
                        Directory to which downloaded files are saved.
  --print-filenames     Print successfully downloaded files's names to stdout.

Results

parfive.Downloader.download returns a parfive.Results object, which is a list of the filenames that have been downloaded. It also tracks any files which failed to download.

Handling Errors

If files fail to download, the urls and the response from the server are stored in the Results object returned by parfive.Downloader. These can be used to inform users about the errors. (Note, the progress bar will finish in an incomplete state if a download fails, i.e. it will show 4/5 Files Downloaded).

The Results object is a list with an extra attribute errors, this property returns a list of named tuples, where these named tuples contains the .url and the .response, which is a aiohttp.ClientResponse or a aiohttp.ClientError object.

Installation

parfive is available on PyPI, you can install it with pip:

pip install parfive

or if you want to use FTP downloads:

pip install parfive[ftp]

Requirements

  • Python 3.9 or above

  • aiohttp

  • tqdm

  • aioftp (for downloads over FTP)

Licence

MIT Licensed

Authors

parfive was written by Stuart Mumford.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parfive-2.3.0.tar.gz (40.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

parfive-2.3.0-py3-none-any.whl (34.6 kB view details)

Uploaded Python 3

File details

Details for the file parfive-2.3.0.tar.gz.

File metadata

  • Download URL: parfive-2.3.0.tar.gz
  • Upload date:
  • Size: 40.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for parfive-2.3.0.tar.gz
Algorithm Hash digest
SHA256 fb21ee82892288e57b4fd8229f746423364b0c4bcaf83bb16ac1e46706f3412b
MD5 06ee7e444329e2844f99f45871987fc9
BLAKE2b-256 89f7d9f27dc57654e9b21e90bed30edee748d56f80e98d2c5a1d5ba1e34c4313

See more details on using hashes here.

Provenance

The following attestation bundles were made for parfive-2.3.0.tar.gz:

Publisher: ci_workflows.yml on Cadair/parfive

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file parfive-2.3.0-py3-none-any.whl.

File metadata

  • Download URL: parfive-2.3.0-py3-none-any.whl
  • Upload date:
  • Size: 34.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for parfive-2.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 95f50cded782b428d166bf8247e07785bdf48d19c8c9c7a0b9474536a5fabc03
MD5 62f92c1e5c94a00ede0fd1ef1e33fe2a
BLAKE2b-256 4ae72a8532ebfc7bff5541a2f0351d9b414e4bb21d4f249cff4cb5452e06279e

See more details on using hashes here.

Provenance

The following attestation bundles were made for parfive-2.3.0-py3-none-any.whl:

Publisher: ci_workflows.yml on Cadair/parfive

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

2.3.1

2 files

This release

2.3.0 This release

2 files

2.2.0

2 files

2.1.0

2 files

2.0.2

2 files

2.0.1

2 files

2.0.0

2 files

1.5.1

2 files

1.5.0

2 files

1.4.0

2 files

1.3.0

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

1 file

1.0.2

1 file

1.0.1

1 file

1.0.0

1 file

0.2.4

1 file

0.2.3

1 file

0.2.2

1 file

0.2.1

1 file

0.2

1 file

0.1.1

1 file

0.1.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page