scrapy-rnet

A Scrapy download handler that routes all HTTP/HTTPS requests through rnet, giving your spiders browser-grade TLS and HTTP/2 fingerprints via BoringSSL impersonation.

Without this, Scrapy uses Python's standard urllib3/twisted stack, which produces a fingerprint trivially identifiable as a bot. With scrapy-rnet, requests look indistinguishable from a real Chrome, Firefox, or Safari browser at the TLS and HTTP/2 layer.

Requirements

• Python 3.13+
• Scrapy 2.14+
• rnet 2.4+

Installation

uv add scrapy-rnet
# or
pip install scrapy-rnet

Setup

Add to your Scrapy settings.py:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http":  "scrapy_rnet.RnetDownloadHandler",
    "https": "scrapy_rnet.RnetDownloadHandler",
}

That's it. All requests will now go through rnet, impersonating Chrome 131 by default.

Configuration

Choosing a browser profile

import rnet

# Chrome (default)
RNET_IMPERSONATE = rnet.Impersonate.Chrome131

# Firefox
RNET_IMPERSONATE = rnet.Impersonate.Firefox133

# Safari
RNET_IMPERSONATE = rnet.Impersonate.Safari18

# Pair with a specific OS fingerprint
RNET_IMPERSONATE    = rnet.Impersonate.Chrome131
RNET_IMPERSONATE_OS = rnet.ImpersonateOS.Windows

Full list of available profiles: rnet docs.

Proxies

Global proxy (all requests):

RNET_PROXIES = ["http://user:pass@proxy.example.com:8080"]

Per-request proxy via request.meta['proxy']:

yield scrapy.Request(url, meta={"proxy": "http://user:pass@proxy.example.com:8080"})

Important: Scrapy's built-in HttpProxyMiddleware strips credentials from the proxy URL before the download handler sees the request. Since rnet manages proxy auth internally, you must disable it:

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
}

How it works

RnetDownloadHandler implements Scrapy's download handler interface (download_request / close). When Scrapy resolves a request for an http or https URL it calls download_request, which:

1. Translates the Scrapy Request (method, URL, headers, body) into an rnet call
2. Sends the request through rnet's BoringSSL-backed async client
3. Converts the rnet response back into the appropriate Scrapy response subclass (HtmlResponse, TextResponse, etc.), preserving status, headers, body, IP address, and HTTP protocol version

A single rnet.Client instance is shared for the spider's lifetime, so connection pooling works as normal.

Testing

# Unit tests only (no network required)
uv run pytest tests/ -m "not integration"

# All tests including real network calls
uv run pytest tests/

The integration suite includes a fingerprint verification test that hits tls.peet.ws and asserts the TLS/HTTP2 fingerprint matches Chrome 131.