cycurl

Documentation

Python binding for curl-impersonate fork via cython. For commercial support, visit impersonate.pro.

curl_cffi\cycurl is the most popular Python binding for curl. Unlike other pure python http clients like httpx or requests, cycurl can impersonate browsers' TLS/JA3 and HTTP/2 fingerprints. If you are blocked by some website for no obvious reason, you can give cycurl a try.

Python 3.9 is the minimum supported version since v0.10.

Sponsors

Maintenance of this project is made possible by all the contributors and sponsors. If you'd like to sponsor this project and have your avatar or company logo appear below click here. 💖

Bypass Cloudflare with API

Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance). Click here to register: https://yescaptcha.com/i/stfnIO

Features

• Supports JA3/TLS and http2 fingerprints impersonation, including recent browsers and custom fingerprints.
• Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
• Mimics the requests API, no need to learn another one.
• Pre-compiled, so you don't have to compile on your machine.
• Supports asyncio with proxy rotation on each request.
• Supports http 2.0 & 3.0, which requests does not.
• Supports websocket.
• MIT licensed.

Notes:

1. For pycurl, you need an http/3 enabled libcurl to make it work, while curl_cffi packages libcurl-impersonate inside Python wheels.
2. Since v0.11.4.

Install

pip install cycurl --upgrade

This should work on Linux, macOS and Windows out of the box. If it does not work on you platform, you may need to compile and install curl-impersonate first and set some environment variables like LD_LIBRARY_PATH.

To install beta releases:

pip install cycurl --upgrade --pre

To install unstable version from GitHub:

git clone https://github.com/synodriver/cycurl/
cd cycurl
python setup.py bdist_wheel

On macOS, you may need to install the following dependencies:

brew install zstd nghttp2

Usage

cycurl comes with a low-level curl API and a high-level requests-like API.

requests-like

<<<<<<< HEAD v0.9:

from cycurl import requests

r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome")

v0.10:

temp

import cycurl

# Notice the impersonate parameter
r = cycurl.get("https://tls.browserleaks.com/json", impersonate="chrome")

print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser

# To keep using the latest browser version as `cycurl` updates,
# simply set impersonate="chrome" without specifying a version.
# Other similar values are: "safari" and "safari_ios"
r = cycurl.get("https://tls.browserleaks.com/json", impersonate="chrome")

# Randomly choose a browser version based on current market share in real world
# from: https://caniuse.com/usage-table
# NOTE: this is a pro feature.
r = cycurl.get("https://example.com", impersonate="realworld")

# To pin a specific version, use version numbers together.
r = cycurl.get("https://tls.browserleaks.com/json", impersonate="chrome124")

# To impersonate other than browsers, bring your own ja3/akamai strings
# See examples directory for details.
r = cycurl.get("https://tls.browserleaks.com/json", ja3=..., akamai=...)

# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = cycurl.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)

proxies = {"https": "socks://localhost:3128"}
r = cycurl.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)

Sessions

<<<<<<< HEAD v0.9:

from cycurl import requests

s = requests.Session()

v0.10:

=======

temp

s = cycurl.Session()

# httpbin is a http test website, this endpoint makes the server set cookies
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>

# retrieve cookies again to verify
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}

Supported impersonate browsers

cycurl supports the same browser versions as supported by the fork of curl-impersonate:

Open source version of cycurl includes versions whose fingerprints differ from previous versions. If you see a version, e.g. chrome135, were skipped, you can simply impersonate it with your own headers and the previous version.

If you don't want to look up the headers etc, by yourself, consider buying commercial support from impersonate.pro, we have comprehensive browser fingerprints database for almost all the browser versions on various platforms.

If you are trying to impersonate a target other than a browser, use ja3=... and akamai=... to specify your own customized fingerprints. See the docs on impersonation for details.

Notes:

1. Added in version 0.6.0.
2. Fixed in version 0.6.0, previous http2 fingerprints were not correct.
3. Added in version 0.7.0.
4. Added in version 0.8.0.
5. Added in version 0.9.0.
6. The version postfix -a(e.g. chrome133a) means that this is an alternative version, i.e. the fingerprint has not been officially updated by browser, but has been observed because of A/B testing.
7. Added in version 0.10.0.
8. Added in version 0.11.0.
9. Since 0.11.0, the format safari184_ios is preferred over safari18_4_ios, both are supported, but the latter is quite confusing and hard to parse.
10. Added in 0.12.0.

Asyncio

from cycurl import AsyncSession

async def main():
    async with AsyncSession() as s:
        r = await s.get("https://example.com")

More concurrency:

import asyncio
from cycurl import AsyncSession

urls = [
    "https://google.com/",
    "https://facebook.com/",
    "https://twitter.com/",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get(url)
        tasks.append(task)
    results = await asyncio.gather(*tasks)

For low-level APIs, Scrapy integration and other advanced topics, see the docs for more details.

WebSockets

from cycurl import WebSocket

def on_message(ws: WebSocket, message: str | bytes):
    print(message)

ws = WebSocket(on_message=on_message)
ws.run_forever("wss://api.gemini.com/v1/marketdata/BTCUSD")

curl-like

Alternatively, you can use the low-level curl-like API:

from cycurl import Curl, CURLOPT_URL, CURLOPT_WRITEDATA
from io import BytesIO

buffer = BytesIO()
c = Curl()
c.setopt(CURLOPT_URL, b'https://tls.browserleaks.com/json')
c.setopt(CURLOPT_WRITEDATA, buffer)

c.impersonate("chrome110")

c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())

See the docs for more details.

scrapy

If you are using scrapy, check out these middlewares:

• tieyongjie/scrapy-fingerprint
• jxlil/scrapy-impersonate
• synodriver/awsome_scrapy_utils

For low-level APIs, Scrapy integration and other advanced topics, see the docs for more details.

Asyncio WebSockets

import asyncio
from cycurl import AsyncSession

async def main():
    async with AsyncSession() as s:
        ws = await s.ws_connect("wss://echo.websocket.org")
        await asyncio.gather(*[ws.send_str("Hello, World!") for _ in range(10)])
        async for message in ws:
            print(message)

Ecosystem

• Integrating with Scrapy: divtiply/scrapy-curl-cffi, jxlil/scrapy-impersonate and tieyongjie/scrapy-fingerprint.
• Integrating with requests, httpx as adapter.
• Integrating with captcha resolvers: CapSolver, YesCaptcha. Please see the head area for promo code and link.

Acknowledgement

• Originally forked from multippt/python_curl_cffi, which is under the MIT license.
• Headers/Cookies files are copied from httpx, which is under the BSD license.
• Asyncio support is inspired by Tornado's curl http client.
• The synchronous WebSocket API is inspired by websocket_client.
• The asynchronous WebSocket API is inspired by aiohttp.

Contributing

When submitting an PR, please use a different branch other than main and check the "Allow edits by maintainers" box, so I can update your PR with lint or style fixes. Thanks!