Curl Adapter

A module that plugs directly into the Python requests library and replaces the default urllib3 HTTP adapter with cURL, equipped with TLS fingerprint-changing capabilities.

Why?

Specifically, this module is meant to be used with the "curl impersonate" python bindings (lexiforest/curl_cffi), in order to send HTTP requests with custom, browser-like TLS & HTTP/2 fingerprints for bypassing sites that detect and block normal python requests (such as Cloudflare for example).

You can also use curl-adapter with pycurl.

Additionally, this module is optimized for seamless integration with Gevent.

Installation

pip install curl-adapter --upgrade --ignore-installed

Usage

Basic example:

import requests
from curl_adapter import CurlCffiAdapter

session = requests.Session()
session.mount("http://", CurlCffiAdapter())
session.mount("https://", CurlCffiAdapter())

# just use requests session like you normally would
session.get("https://example.com")

Configuring curl impersonate options:

import requests
from curl_adapter import CurlCffiAdapter

curl_cffi_adapter = CurlCffiAdapter(
    # This is the default
    impersonate_browser_type="chrome", 

    # Optionally set additional options
    tls_configuration_options={
        "ja3_str": "...",
        "akamai_str": "...",
        "extra_fp": ExtraFingerprints(...),
    }
)

# you can use 'with ...' for just making a single request
with requests.Session() as s:
    s.mount("http://", curl_cffi_adapter)
    s.mount("https://", curl_cffi_adapter)

    s.get("https://example.com")

Using it with pycurl:

import requests
from curl_adapter import PyCurlAdapter

with requests.Session() as s:
    s.mount("http://", PyCurlAdapter())
    s.mount("https://", PyCurlAdapter())

    s.get("https://example.com")

More

You can get extra information from the curl response info:

import requests
from curl_adapter import PyCurlAdapter, get_curl_info

with requests.Session() as s:
    s.mount("http://", PyCurlAdapter())
    s.mount("https://", PyCurlAdapter())

    response = s.get("https://example.com")

    body = response.text

    curl_info = get_curl_info(response)

    print(
        curl_info
    )

Returns a simple dict:

{
    'local_ip':'192.168.1.1',
    'local_port':19219,
    'primary_ip':'142.250.200.142',
    'primary_port':443,
    'request_size':0,
    'request_body_size':0,
    'response_header_size':418,
    'ssl_verify_result':0,
    'proxy_ssl_verify_result':0,
    'starttransfer_time':171335,
    'connect_time':33231,
    'appconnect_time':47274,
    'pretransfer_time':47378,
    'namelookup_time':1025,
    'has_used_proxy':0,
    'speed_download':52081115, # only available after the body has been read
    'speed_upload':0, # only available after the body has been read
    'response_body_size':519958376, # only available after the body has been read
    'total_time':9983626, # only available after the body has been read
}

Note that some cURL information fields are only availabe after the body stream has been fully consumed, so keep that in mind when using stream=True option.