scrapy-download-handlers-incubator
Overview
This is a collection of semi-official download handlers for Scrapy. See the Scrapy download handler documentation for more information.
They should work and some of them may be later promoted to the official status, but here they are provided as-is and no support or stability promises are given. The documentation, including limitations and unsupported features, is also provided as-is and may be incomplete.
As this code intentionally uses private Scrapy APIs, it specifies a tight dependency on Scrapy. This version of the package only supports Scrapy 2.19.x.
Features overview
The baseline for these handlers is the default Scrapy handler, HTTP11DownloadHandler, which uses Twisted and supports HTTP/1.1. Feature parity with it is an explicit goal but it’s not always possible and not all possible features are implemented in all handlers (which may change in the future). Certain popular features not supported by HTTP11DownloadHandler, like HTTP/2 support, and features unique to some handlers, may or may not be implemented. Please see the sections for individual handlers for more details.
The following table summarizes the most important differences:
Handler |
HTTP/2 |
Proxies |
Impersonation |
TLS implementation |
|---|---|---|---|---|
(HTTP11DownloadHandler) |
Not possible |
Yes |
Not possible |
cryptography |
AiohttpDownloadHandler |
Not possible |
Yes |
Not possible |
Stdlib ssl |
CurlCffiDownloadHandler |
Yes |
Yes |
Yes |
libcurl |
HttpxDownloadHandler |
Yes |
Yes |
Not possible |
Stdlib ssl |
NiquestsDownloadHandler |
Yes |
Yes |
Not possible |
Stdlib ssl |
PyreqwestDownloadHandler |
Yes |
Not possible |
Not possible |
rustls |
WreqDownloadHandler |
Yes |
Yes |
Yes |
BoringSSL |
The following basic features are supported by all handlers unless mentioned in their docs:
Native asyncio integration without requiring a Twisted reactor
HTTP/1.1 for http and https schemes
Unified download handler exceptions
Proxies, including HTTP and HTTPS proxies for HTTP and HTTPS destinations
Proxy authentication via HttpProxyMiddleware
IPv6 destinations
DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE and the respective request meta keys
DOWNLOAD_TIMEOUT and the respective request meta key
DOWNLOAD_FAIL_ON_DATALOSS and the "dataloss" flag
Setting the download_latency request meta
DOWNLOAD_BIND_ADDRESS
DOWNLOAD_VERIFY_CERTIFICATES
headers_received and bytes_received signals
Not reading the proxy configuration from the environment variables
Not handling cookies, redirects, compression and other things handled by Scrapy itself
Handlers
AiohttpDownloadHandler
This is an updated copy of the official scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler handler. It supports HTTP/1.1 and uses the aiohttp library.
Install it with:
pip install scrapy-download-handlers-incubatorEnable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.AiohttpDownloadHandler",
"https": "scrapy_download_handlers_incubator.AiohttpDownloadHandler",
}Features and limitations
HTTP proxies |
Yes |
SOCKS proxies |
No (not supported by the library) |
HTTP/2 |
No (not supported by the library) |
TLS verbose logging |
Yes |
response.ip_address |
Yes |
response.certificate |
Yes (DER bytes) |
Per-request bindaddress |
No (not supported by the library) |
Proxy certificate verification |
Follows DOWNLOAD_VERIFY_CERTIFICATES |
TLS implementation |
Standard library ssl |
Notable features supported by the library but not implemented:
DNS resolving settings
Custom DNS resolvers
CurlCffiDownloadHandler
This handler supports HTTP/1.1 and HTTP/2 and uses the curl_cffi library. It also supports impersonating TLS, HTTP/2 and header fingerprints of browsers.
Install it with:
pip install scrapy-download-handlers-incubator[curl-cffi]Enable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.CurlCffiDownloadHandler",
"https": "scrapy_download_handlers_incubator.CurlCffiDownloadHandler",
}Features and limitations
HTTP proxies |
Yes |
SOCKS proxies |
Yes (SOCKS4, SOCKS5) |
HTTP/2 |
Yes |
HTTP/3 |
Yes (but not tested) |
TLS verbose logging |
No (not supported by the library) |
response.ip_address |
Yes |
response.certificate |
No (not supported by the library) |
Per-request bindaddress |
No (not supported by the library) |
Proxy certificate verification |
Follows DOWNLOAD_VERIFY_CERTIFICATES |
TLS implementation |
libcurl |
Notable features supported by the library but not implemented:
Advanced libcurl tunables
Settings
CURL_CFFI_HTTP_VERSION (str, default: "v1", corresponding to “Enforce HTTP/1.1”): The HTTP version to use. The value is passed directly to the library so the possible values are set by curl_cffi.requests.utils.normalize_http_version() and the meanings of the underlying constants can be seen in libcurl docs (CURLOPT_HTTP_VERSION). Set this to "v2tls" or "v2" to enable HTTP/2 for HTTPS requests or for all requests respectively. Set this to "v3" to enable HTTP/3.
CURL_CFFI_IMPERSONATE (str, dict or curl_cffi.Fingerprint, default: None): The browser to impersonate. Either a target name supported by curl_cffi, e.g. "chrome" or "firefox135", or a custom fingerprint as a curl_cffi.Fingerprint object or a dict of its arguments.
Request meta keys
curl_cffi_impersonate: Same as CURL_CFFI_IMPERSONATE, overrides it for this request.
Impersonation details
With HTTP/2 disabled the TLS fingerprint won’t match the one of the impersonated browser, so you should also set CURL_CFFI_HTTP_VERSION to "v2tls".
Impersonation adds browser request headers, but headers set by Scrapy override them. To send the impersonated User-Agent, Accept and Accept-Language headers, disable the ones set by Scrapy:
USER_AGENT = None
DEFAULT_REQUEST_HEADERS = {}HttpxDownloadHandler
This is an updated copy of the official scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler handler. It supports HTTP/1.1 and HTTP/2 and uses the httpx2 library.
Install it with:
pip install scrapy-download-handlers-incubator[httpx]Enable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.HttpxDownloadHandler",
"https": "scrapy_download_handlers_incubator.HttpxDownloadHandler",
}Features and limitations
HTTP proxies |
Yes |
SOCKS proxies |
Yes (SOCKS5; requires httpx2[socks]) |
HTTP/2 |
Yes (requires httpx2[http2]) |
HTTP/3 |
No (not supported by the library) |
TLS verbose logging |
Yes |
response.ip_address |
Yes |
response.certificate |
Yes (DER bytes) |
Per-request bindaddress |
No (not supported by the library) |
Proxy certificate verification |
Follows DOWNLOAD_VERIFY_CERTIFICATES |
TLS implementation |
Standard library ssl |
Other limitations:
The handler creates a separate connection pool for each proxy URL (due to limitations of httpx2) which may lead to higher resource usage when using proxy rotation.
Notable features supported by the library but not implemented:
Alternative transports
Limiting the number of per-proxy connection pool to save resources
Settings
HTTPX_HTTP2_ENABLED (bool, default: False): Whether to enable HTTP/2.
NiquestsDownloadHandler
This handler supports HTTP/1.1 and HTTP/2 and uses the niquests library.
Install it with:
pip install scrapy-download-handlers-incubator[niquests]Enable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.NiquestsDownloadHandler",
"https": "scrapy_download_handlers_incubator.NiquestsDownloadHandler",
}Features and limitations
HTTP proxies |
Yes |
SOCKS proxies |
Yes (SOCKS4, SOCKS5; requires niquests[socks]) |
HTTP/2 |
Yes |
HTTP/3 |
No (not implemented) |
TLS verbose logging |
Yes |
response.ip_address |
Yes |
response.certificate |
Yes (DER bytes) |
Per-request bindaddress |
No (not supported by the library) |
Proxy certificate verification |
Follows DOWNLOAD_VERIFY_CERTIFICATES |
TLS implementation |
Standard library ssl |
Notable features supported by the library but not implemented:
Custom DNS resolvers
HTTP/2 tunables
Settings
NIQUESTS_HTTP2_ENABLED (bool, default: False): Whether to enable HTTP/2.
PyreqwestDownloadHandler
This handler supports HTTP/1.1 and HTTP/2 and uses the pyreqwest library.
Install it with:
pip install scrapy-download-handlers-incubator[pyreqwest]Enable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.PyreqwestDownloadHandler",
"https": "scrapy_download_handlers_incubator.PyreqwestDownloadHandler",
}Features and limitations
Proxies |
No (not supported by the library) |
HTTP/2 |
Yes |
HTTP/3 |
No (not supported by the library) |
TLS verbose logging |
No (not supported by the library) |
response.ip_address |
No (not supported by the library) |
response.certificate |
No (not supported by the library) |
Per-request bindaddress |
No (not supported by the library) |
TLS implementation |
Rust rustls |
Notable features supported by the library but not implemented:
HTTP/2 tunables
Settings
PYREQWEST_HTTP2_ENABLED (bool, default: False): Whether to enable HTTP/2.
WreqDownloadHandler
This handler supports HTTP/1.1 and HTTP/2 and uses the wreq library. It also supports emulating TLS, HTTP/2 and header fingerprints of browsers and other clients.
Install it with:
pip install scrapy-download-handlers-incubator[wreq]Enable it with:
DOWNLOAD_HANDLERS = {
"http": "scrapy_download_handlers_incubator.WreqDownloadHandler",
"https": "scrapy_download_handlers_incubator.WreqDownloadHandler",
}Features and limitations
HTTP proxies |
Yes |
SOCKS proxies |
Yes (SOCKS4, SOCKS5) |
HTTP/2 |
Yes |
HTTP/3 |
No (not supported by the library) |
TLS verbose logging |
No (not supported by the library) |
response.ip_address |
Yes |
response.certificate |
Yes (DER bytes) |
Per-request bindaddress |
Yes |
Proxy certificate verification |
Follows DOWNLOAD_VERIFY_CERTIFICATES |
TLS implementation |
BoringSSL |
Notable features supported by the library but not implemented:
DNS resolving settings
TLS and HTTP/2 tunables
Client certificates
Settings
WREQ_HTTP2_ENABLED (bool, default: False): Whether to enable HTTP/2.
WREQ_EMULATION (str or dict, default: None): The client to emulate. Either a name of a wreq.Profile member, e.g. "Chrome149", or a dict of wreq.Emulation arguments, with names of wreq.Profile and wreq.Platform members as the profile and platform values, e.g. {"profile": "Chrome149", "platform": "Windows"}.
Request meta keys
wreq_emulation: Same as WREQ_EMULATION, overrides it for this request.
Emulation details
With HTTP/2 disabled the TLS fingerprint won’t match the one of the emulated browser, so you should also set WREQ_HTTP2_ENABLED to True.
Emulation adds browser request headers, but headers set by Scrapy override them. To send the emulated User-Agent, Accept and Accept-Language headers, disable the ones set by Scrapy:
USER_AGENT = None
DEFAULT_REQUEST_HEADERS = {}1 maintainer
Metadata
1 maintainer
Release files for scrapy-download-handlers-incubator 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapy_download_handlers_incubator-0.7.0.tar.gz | 68.8 kB | With provenance | Details for scrapy_download_handlers_incubator-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scrapy_download_handlers_incubator-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for scrapy_download_handlers_incubator-0.7.0-py3-none-any.whl |
Total release size: 96.7 kB
Release files / scrapy_download_handlers_incubator-0.7.0.tar.gz
| Download URL | scrapy_download_handlers_incubator-0.7.0.tar.gz |
|---|---|
| Size | 68.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fcdcb3b01362464d1350065af3bca6a7590a9c63aad4ab6c7e0e039835254e4c
|
|
BLAKE2b-256 checksum How to use checksums |
a084baedd6aca45a51bf5f72db9fa8f9777d2103a472b878665dd2a92979585e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logRelease files / scrapy_download_handlers_incubator-0.7.0-py3-none-any.whl
| Download URL | scrapy_download_handlers_incubator-0.7.0-py3-none-any.whl |
|---|---|
| Size | 27.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
76599d472a17b018ea2e44c8bd1f15ae8e5aa10a45c245756601e5ed01e59d30
|
|
BLAKE2b-256 checksum How to use checksums |
272f1761899d88d5a090847028454284e9e542fec8b91fc9cec6f1955d6827d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logSecurity for scrapy-download-handlers-incubator 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for scrapy_download_handlers_incubator-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
