scrapy-impersonate

scrapy-impersonate is a Scrapy download handler. This project integrates curl_cffi to perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.

Installation

pip install scrapy-impersonate

Activation

To use this package, replace the default http and https Download Handlers by updating the DOWNLOAD_HANDLERS setting:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

By setting USER_AGENT = None, curl_cffi will automatically choose the appropriate User-Agent based on the impersonated browser:

USER_AGENT = ""

Also, be sure to install the asyncio-based Twisted reactor for proper asynchronous execution:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Usage

Set the impersonate Request.meta key to download a request using curl_cffi:

import scrapy


class ImpersonateSpider(scrapy.Spider):
    name = "impersonate_spider"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "USER_AGENT": "",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
    }

    def start_requests(self):
        for _ in range(5):
            yield scrapy.Request(
                "https://tls.browserleaks.com/json",
                dont_filter=True,
            )

    def parse(self, response):
        # ja3_hash: 98cc085d47985d3cca9ec1415bbbf0d1 (chrome133a)
        # ja3_hash: 2d692a4485ca2f5f2b10ecb2d2909ad3 (firefox133)
        # ja3_hash: c11ab92a9db8107e2a0b0486f35b80b9 (chrome124)
        # ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37 (safari15_5)
        # ja3_hash: cd08e31494f9531f560d64c695473da9 (chrome99_android)

        yield {"ja3_hash": response.json()["ja3_hash"]}

impersonate-args

You can pass any necessary arguments to curl_cffi through impersonate_args. For example:

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": browser,
        "impersonate_args": {
            "verify": False,
            "timeout": 10,
        },
    },
)

Some arguments worth knowing about:

[!WARNING] allow_redirects is forced to False so that redirects are handled by Scrapy. Re-enabling it through impersonate_args makes curl_cffi follow redirects internally, which bypasses Scrapy's redirect and offsite middlewares and exposes you to redirect-based SSRF.

curl-options

For settings that have no curl_cffi argument, impersonate_curl_options passes raw libcurl options through. Keys can be CurlOpt members or their names, and they take precedence over the options set by this handler.

The most common use is pinning the header order, which browsers keep stable and WAFs check (HTTPHEADER_ORDER requires curl_cffi >= 0.16.0):

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": "chrome",
        "impersonate_curl_options": {
            "HTTPHEADER_ORDER": "Host,Connection,User-Agent,Accept,Referer",
        },
    },
)

Supported browsers

The following browsers can be impersonated (curl_cffi >= 0.15.0):

Browser	Version	OS	Name	HTTP/3
	99	Windows 10	chrome99
	99	Android 12	chrome99_android
	100	Windows 10	chrome100
	101	Windows 10	chrome101
	104	Windows 10	chrome104
	107	Windows 10	chrome107
	110	Windows 10	chrome110
	116	Windows 10	chrome116
	119	macOS Sonoma	chrome119
	120	macOS Sonoma	chrome120
	123	macOS Sonoma	chrome123
	124	macOS Sonoma	chrome124
	131	macOS Sonoma	chrome131
	131	Android 14	chrome131_android
	133	macOS Sequoia	chrome133a
	136	macOS Sequoia	chrome136
	142	macOS Tahoe	chrome142
	145	macOS Tahoe	chrome145	✅
	146	macOS Tahoe	chrome146	✅
	99	Windows 10	edge99
	101	Windows 10	edge101
	15.3	macOS Big Sur	safari153
	15.5	macOS Monterey	safari155
	17.0	macOS Sonoma	safari170
	17.2	iOS 17.2	safari172_ios
	18.0	macOS Sequoia	safari180
	18.0	iOS 18.0	safari180_ios
	18.4	macOS Sequoia	safari184
	18.4	iOS 18.4	safari184_ios
	26.0	macOS Tahoe	safari260
	26.0	iOS 26.0	safari260_ios
	26.0.1	macOS Tahoe	safari2601
	133.0	macOS Sonoma	firefox133
	135.0	macOS Sonoma	firefox135
	144.0	macOS Tahoe	firefox144
	147.0	macOS Tahoe	firefox147	✅
	14.5	macOS Sonoma	tor145

_Notes:

1. The old Safari target names (safari15_3, safari15_5, safari17_0, safari17_2_ios, safari18_0, safari18_0_ios) are kept as deprecated aliases. Prefer the new names (safari153, safari155, …) for new code.
2. You can also pass a floating value like impersonate="chrome", "safari", "safari_ios" or "firefox" to let curl_cffi pick the most recent fingerprint without pinning a version.
3. RandomBrowserMiddleware rotates across chrome, firefox, safari, edge and tor by default. Override with IMPERSONATE_BROWSERS to narrow the set (e.g. IMPERSONATE_BROWSERS = ["chrome", "firefox"]).

Development

Install the development dependencies and run the test suite:

pip install -r requirements-dev.txt
pip install -e .
pytest

The tests spin up local HTTP/HTTPS servers and a CONNECT proxy, so no network access is required.

Thanks

This project is inspired by the following projects:

• curl_cffi - Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
• curl-impersonate - A special build of curl that can impersonate Chrome & Firefox
• scrapy-playwright - Playwright integration for Scrapy