fastdl 0.1

Download and extract files fast and easily

fastdl

Download and extract files fast and easily.

file_path = fastdl.download(url, extract=True, dir_prefix="downloads")

Features

• Parallel downloads (threads and processes)
• Cache for downloads and extractions
• Easy extractions
• Progress bars
• Easily configurable

Installation

$ pip install fastdl

Usage

fastdl can be be used in two different ways, download a single file or download multiple files in parallel:

Single file

file_path = fastdl.download(url, dir_prefix="downloads", extract=True)

Multiple files

with fastdl.Parallel(prefer="threads"):
    downloads = []

    for url in urls:
        download = p.download(url, dir_prefix="downloads") # same arguments

    for download in downloads:
        file_path = download.get()  # block until download is finished

Parameters

Parameter	Description	Type	Default
url	Url to download	str
fname	File name of download file. Can be an absolute path. Can receive a function with response as an argument	str or callable	None
dir_prefix	Directory to store download file. Used only if fname is not an absolute path. Defaults to current directory or value specified by fastdl.conf["default_dir_prefix"]	str	None
subdir_prefix	Subdirectory inside dir_prefix to store download. Defaults to no subdirectory.	str	""
headers	Dictionnary of HTTP headers to send. For example: {"User-Agent": "fastdl/0.1"}	dict	{}
blocksize	Number of bytes to read and write for each iteration	int	8192
file_hash	File hash. If the file hash doesn't match, it will be re-downloaded.	str	None
hash_algorithm	Hash algorithm. One of the following: "sha256", "sha512", "sha1", "md5" or auto. By default, it will try to infer the algorithm according to the length of the file_hash	str	auto
extract	Whether or not extract file	bool	False
extract_dir	Directory to store extracted files. By default same directory as the file	str	None
progressbar	Whether or not show a progress bar	bool	True
force_download	Whether or not force download if file already exists. By default, it doesn't re-download file unless file hash doesn't match	bool	False
force_extraction	Whether or not force extraction if file already exists	bool	False

Examples

Download file and re-download if hash doesn't match

import fastdl

file_hash = "155fdb3732e82cc4864c441e6400def0"
url = "https://file-examples.com/wp-content/uploads/2017/02/zip_2MB.zip"

file_path = fastdl.download(url, extract=True, dir_prefix="downloads", file_hash=file_hash)

Download multiple files in parallel

import fastdl

urls = [
    "https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_480_1_5MG.mp4",
    "https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_640_3MG.mp4",
    "https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1280_10MG.mp4"
]

with fastdl.Parallel(prefer="processes", max_workers=2) as p:
    downloads = []

    for url in urls:
        download = p.download(url, dir_prefix="downloads")

    for download in downloads:
        file_path = download.get(timeout=10)  # wait 10 seconds or raise timeout if download hasn't finished

Download from a list of urls in parallel

import fastdl
from fastdl.utils import lines

with fastdl.Parallel() as p:
    downloads = []

    for url in lines("urls.txt"):
        download = p.download(url)
        downloads.append(download)

    for download in downloads:
        file_path = download.get()

Change default download directory

If you're using a directory for all your downloads (e.g ~/.myapp), you can easily change the default directory for your downloads:

# myapp/__init__.py

import fastdl

fastdl.conf["default_dir_prefix"] = "~/.myapp"

# myapp/file.py

file_path = fastdl.download("https://file-examples.com/wp-content/uploads/2017/02/zip_2MB.zip")
file_path == "~/.myapp/zip_2MB.zip"

Check examples folder to see examples in action.