Flardl
Project description
Flardl - Adaptive Multi-Site Downloading of Lists
Who would flardls bear?
Features
Flardl adaptively downloads a list of files from a list of federated web servers. Federated, in this case, means that one can download the same file from each server in the list. For lists of a few hundred or more files of ~1MB in size, the download speed can approach Gbit/s line limits, typically 300X higher than a curl-based script.
The main speed-up is obtained by asynchronous I/O; the use of multiple servers provides stability and dynamic adaptability in the face of unknown server loads and net weather.
The name flardl could be either an acronym involving downloading, or a nonsense word. You pick.
Requirements
Flardl is tested under python 3.9 to 3.11, on Linux, MacOS, and Windows. Under the hood, flardl relies on httpx and is supported on whatever platforms that library works under, for both HTTP/1.1 and HTTP/2. HTTP/3 support could easily be added via aioquic once enough servers are running HTTP/3 to make that worthwhile.
Installation
You can install Flardl via pip from PyPI:
$ pip install flardl
Usage
Please see the [Command-line Reference] for details.
Contributing
Contributions are very welcome. To learn more, see the Contributor Guide.
License
Distributed under the terms of the BSD 3-clause_license, Flardl is free and open source software.
Issues
If you encounter any problems, please file an issue along with a detailed description.
Credits
Flardl was written by Joel Berendzen.
This project was generated from @cjolowicz's Hypermodern Python Cookiecutter template.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.