Overview

This is a collection of semi-official download handlers for Scrapy. See the Scrapy download handler documentation for more information.

They should work and some of them may be later promoted to the official status, but here they are provided as-is and no support or stability promises are given. The documentation, including limitations and unsupported features, is also provided as-is and may be incomplete.

As this code intentionally uses private Scrapy APIs, it specifies a tight dependency on Scrapy. This version of the package only supports Scrapy 2.19.x.

Features overview

The baseline for these handlers is the default Scrapy handler, HTTP11DownloadHandler, which uses Twisted and supports HTTP/1.1. Feature parity with it is an explicit goal but it’s not always possible and not all possible features are implemented in all handlers (which may change in the future). Certain popular features not supported by HTTP11DownloadHandler, like HTTP/2 support, and features unique to some handlers, may or may not be implemented. Please see the sections for individual handlers for more details.

The following table summarizes the most important differences:

The following basic features are supported by all handlers unless mentioned in their docs:

• Native asyncio integration without requiring a Twisted reactor

• HTTP/1.1 for http and https schemes

• Unified download handler exceptions

• Proxies, including HTTP and HTTPS proxies for HTTP and HTTPS destinations

• Proxy authentication via HttpProxyMiddleware

• IPv6 destinations

• DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE and the respective request meta keys

• DOWNLOAD_TIMEOUT and the respective request meta key

• DOWNLOAD_FAIL_ON_DATALOSS and the "dataloss" flag

• Setting the download_latency request meta

• DOWNLOAD_BIND_ADDRESS

• DOWNLOAD_VERIFY_CERTIFICATES

• headers_received and bytes_received signals

• Not reading the proxy configuration from the environment variables

• Not handling cookies, redirects, compression and other things handled by Scrapy itself

Handlers

pip install scrapy-download-handlers-incubator

DOWNLOAD_HANDLERS = {
    "http": "scrapy_download_handlers_incubator.AiohttpDownloadHandler",
    "https": "scrapy_download_handlers_incubator.AiohttpDownloadHandler",
}

pip install scrapy-download-handlers-incubator[curl-cffi]

DOWNLOAD_HANDLERS = {
    "http": "scrapy_download_handlers_incubator.CurlCffiDownloadHandler",
    "https": "scrapy_download_handlers_incubator.CurlCffiDownloadHandler",
}

pip install scrapy-download-handlers-incubator[httpx]

DOWNLOAD_HANDLERS = {
    "http": "scrapy_download_handlers_incubator.HttpxDownloadHandler",
    "https": "scrapy_download_handlers_incubator.HttpxDownloadHandler",
}

pip install scrapy-download-handlers-incubator[niquests]

DOWNLOAD_HANDLERS = {
    "http": "scrapy_download_handlers_incubator.NiquestsDownloadHandler",
    "https": "scrapy_download_handlers_incubator.NiquestsDownloadHandler",
}

pip install scrapy-download-handlers-incubator[pyreqwest]

DOWNLOAD_HANDLERS = {
    "http": "scrapy_download_handlers_incubator.PyreqwestDownloadHandler",
    "https": "scrapy_download_handlers_incubator.PyreqwestDownloadHandler",
}