scrapy-curl-cffi

Scrapy integration with curl_cffi (curl-impersonate).

Installation

pip install scrapy-curl-cffi

Another option, to enable Scrapy's support for modern HTTP compression protocols:

pip install scrapy-curl-cffi[compression]

Configuration

Update your Scrapy project settings as follows:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_curl_cffi.handlers.CurlCffiDownloadHandler",
    "https": "scrapy_curl_cffi.handlers.CurlCffiDownloadHandler",
}

DOWNLOADER_MIDDLEWARES = {
    "scrapy_curl_cffi.middlewares.CurlCffiMiddleware": 200,
    "scrapy_curl_cffi.middlewares.DefaultHeadersMiddleware": 400,
    "scrapy_curl_cffi.middlewares.UserAgentMiddleware": 500,
    "scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware": None,
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Usage

To download a scrapy.Request with curl_cffi, add the curl_cffi_options special key to the Request.meta attribute. The value should be a dict with any of the following options:

• impersonate - which browser version to impersonate
• ja3 - ja3 string to impersonate
• akamai - akamai string to impersonate
• extra_fp - extra fingerprints options, in complement to ja3 and akamai strings
• default_headers - whether to set default browser headers when impersonating, defaults to True
• verify - whether to verify https certs, defaults to the DOWNLOAD_VERIFY_CERTIFICATES setting
• http_version - HTTP version to force, e.g. "v1", "v2" or "v3"; defaults to the CURL_CFFI_HTTP_VERSION setting

See the curl_cffi documentation for more info on these options.

Alternatively, you can use the curl_cffi_options spider attribute or the CURL_CFFI_OPTIONS setting to automatically assign the curl_cffi_options meta for all requests.

Example spider

class FingerprintsSpider(scrapy.Spider):
    name = "fingerprints"
    start_urls = ["https://tls.browserleaks.com/json"]
    curl_cffi_options = {"impersonate": "chrome"}

    def parse(self, response):
        yield response.json()

curl_cffi interop

scrapy-curl-cffi strives to adhere to established Scrapy conventions, ensuring that most Scrapy settings, spider attributes, request/response attributes and meta keys configure the crawler's behavior in an expected manner.

All curl_cffi downloads share one curl_cffi.AsyncSession, created lazily on the first curl_cffi download and reused for connection pooling. The following settings configure it:

• CURL_CFFI_MAX_CLIENTS - hard limit on simultaneous curl_cffi connections, defaults to 10
• DOWNLOAD_VERIFY_CERTIFICATES - whether to verify https certs, as in Scrapy itself
• CURL_CFFI_HTTP_VERSION - HTTP version to force, e.g. "v1", "v2" or "v3"; unset lets curl_cffi (and browser impersonation) pick the version
• DOWNLOAD_BIND_ADDRESS - local address to bind curl_cffi downloads to, as in Scrapy itself; the bindaddress request meta key overrides it
• DOWNLOAD_TLS_MIN_VERSION / DOWNLOAD_TLS_MAX_VERSION - TLS version limits, as in Scrapy itself

Similar projects

• scrapy-impersonate
• scrapy-fingerprint