PyURLFetch

A Python library for implementing convenient resumable download functionality.

Extracted from MacWave's package download flow. It handles .partial files and HTTP Range requests for resuming, optional rate limiting, proxies, skipping SSL verification, and a rich progress bar.

Install

Not published on PyPI yet. Install name is PyURLFetch, import name is pyurlfetch — install straight from the repository:

# core (requests + urllib3)
pip install "git+https://github.com/Sha0huaZhang/PyURLFetch.git"

# with the rich progress bar
pip install "pyurlfetch[progress] @ git+https://github.com/Sha0huaZhang/PyURLFetch.git"

Or from a local clone:

git clone https://github.com/Sha0huaZhang/PyURLFetch.git
cd PyURLFetch
pip install .

Usage

from pyurlfetch import download_file

download_file(
    "https://example.com/archive.tar.gz",
    "archive.tar.gz",
    resume=True,
    limit_rate="1M",
)

The download is written to archive.tar.gz.partial and renamed to the target once complete, so an interrupted download can be resumed by calling the same function again with resume=True.

Failures raise an exception instead of exiting the process:

from pyurlfetch import HTTPStatusError, NetworkError, UserAborted

try:
    download_file(url, dest, resume=True)
except HTTPStatusError as error:
    print(error.status_code, error.url)
except (NetworkError, UserAborted) as error:
    print("download failed:", error)

Retrying

download_file does not prompt interactively. Control retries either with max_retries or with an on_retry(attempt, error) -> bool callback:

download_file(url, dest, max_retries=3)

download_file(url, dest, on_retry=lambda attempt, error: input("retry? [y/N] ") == "y")