pooch-rattler 0.3.0

Pooch downloader powered by Rattler

pooch-rattler

Pooch downloader powered by Rattler.

Features

• Native Rust performance
• Work with anyio, asyncio and trio
• Drop-in replacement of the global default Pooch downloader
• Reqwest middlewares which bring the missing GCS, S3 and retry support in Pooch

Get started

import anyio.to_thread
import pooch
import pooch_rattler

# Register global default
pooch_rattler.install()

# Synchronous download
downloaded_file = pooch.retrieve("https://example.com/index.html")

# Asynchronous download
async def async_download():
    # We still need a worker thread to look for existing cache,
    # validate checksums and decompress downloaded files
    return await anyio.to_thread.run_sync(
        pooch.retrieve, "https://example.com/index.html")

cached_file = anyio.run(async_download)
assert cached_file == downloaded_file

Downloaders can also be created and invoked explicitly. Examples can be found in the unit tests.

Resumable downloader

Resumable downloader provides functionality like curl -C - (man page). It's not the default behavior due to the following limitations:

• The server must know how to handle HTTP range requests, otherwise the downloaded file can be corrupted since there is no way to read response headers into Python from Rattler.
• Resumable downloader also introduces slightly more Python overhead even if no actual resumption takes place.

Resumable downloader supports most of the features described in the previous sections. To use it, replace your pooch_rattler.Downloader with pooch_rattler.ResumableDownloader.