Skip to main content

libcurl ffi bindings for Python, with impersonation support.

Project description

curl_cffi

PyPI - Downloads PyPI - Python Version PyPI version Generic badge Generic badge

Documentation

Python binding for curl-impersonate fork via cffi. For commercial support, visit impersonate.pro.

curl_cffi is the most popular Python binding for curl. Unlike other pure python http clients like httpx or requests, curl_cffi can impersonate browsers' TLS/JA3 and HTTP/2 fingerprints. If you are blocked by some website for no obvious reason, you can give curl_cffi a try.

Python 3.9 is the minimum supported version since v0.10.

Sponsors

Maintenance of this project is made possible by all the contributors and sponsors. If you'd like to sponsor this project and have your avatar or company logo appear below click here. 💖


SerpAPI

Scrape Google and other search engines from SerpApi's fast, easy, and complete API. 0.66s average response time (≤ 0.5s for Ludicrous Speed Max accounts), 99.95% SLAs, pay for successful responses only.


Bypass Cloudflare with API

Yes Captcha!

Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance). Click here to register: https://yescaptcha.com/i/stfnIO


Easy Captcha Bypass for Scraping

CapSolver

CapSolver is an AI-powered tool that easily bypasses Captchas, allowing uninterrupted access to public data. It supports a variety of Captchas and works seamlessly with curl_cffi, Puppeteer, Playwright, and more. Fast, reliable, and cost-effective. Plus, curl_cffi users can use the code "CURL" to get an extra 6% balance! and register here.


Features

  • Supports JA3/TLS and http2 fingerprints impersonation, including recent browsers and custom fingerprints.
  • Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
  • Mimics the requests API, no need to learn another one.
  • Pre-compiled, so you don't have to compile on your machine.
  • Supports asyncio with proxy rotation on each request.
  • Supports http 2.0 & 3.0, which requests does not.
  • Supports websocket.
  • MIT licensed.
requests aiohttp httpx pycurl curl_cffi
http/2
http/3 ☑️1 2
sync
async
websocket
fingerprints
speed 🐇 🐇🐇 🐇 🐇🐇 🐇🐇

Notes:

  1. For pycurl, you need an http/3 enabled libcurl to make it work, while curl_cffi packages libcurl-impersonate inside Python wheels.
  2. Since v0.11.4.

Install

pip install curl_cffi --upgrade

This should work on Linux, macOS and Windows out of the box. If it does not work on you platform, you may need to compile and install curl-impersonate first and set some environment variables like LD_LIBRARY_PATH.

To install beta releases:

pip install curl_cffi --upgrade --pre

To install unstable version from GitHub:

git clone https://github.com/lexiforest/curl_cffi/
cd curl_cffi
make preprocess
pip install .

On macOS, you may need to install the following dependencies:

brew install zstd nghttp2

Usage

curl_cffi comes with a low-level curl API and a high-level requests-like API.

requests-like

v0.9:

from curl_cffi import requests

r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome")

v0.10:

import curl_cffi

# Notice the impersonate parameter
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome")

print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser

# To keep using the latest browser version as `curl_cffi` updates,
# simply set impersonate="chrome" without specifying a version.
# Other similar values are: "safari" and "safari_ios"
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome")

# Randomly choose a browser version based on current market share in real world
# from: https://caniuse.com/usage-table
# NOTE: this is a pro feature.
r = curl_cffi.get("https://example.com", impersonate="realworld")

# To pin a specific version, use version numbers together.
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome124")

# To impersonate other than browsers, bring your own ja3/akamai strings
# See examples directory for details.
r = curl_cffi.get("https://tls.browserleaks.com/json", ja3=..., akamai=...)

# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)

proxies = {"https": "socks://localhost:3128"}
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)

Sessions

v0.9:

from curl_cffi import requests

s = requests.Session()

v0.10:

s = curl_cffi.Session()

# httpbin is a http test website, this endpoint makes the server set cookies
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>

# retrieve cookies again to verify
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}

Supported impersonate browsers

curl_cffi supports the same browser versions as supported by my fork of curl-impersonate:

Open source version of curl_cffi includes versions whose fingerprints differ from previous versions. If you see a version, e.g. chrome135, were skipped, you can simply impersonate it with your own headers and the previous version.

If you don't want to look up the headers etc, by yourself, consider buying commercial support from impersonate.pro, we have comprehensive browser fingerprints database for almost all the browser versions on various platforms.

If you are trying to impersonate a target other than a browser, use ja3=... and akamai=... to specify your own customized fingerprints. See the docs on impersonation for details.

Browser Open Source Pro version
Chrome chrome99, chrome100, chrome101, chrome104, chrome107, chrome110, chrome116[1], chrome119[1], chrome120[1], chrome123[3], chrome124[3], chrome131[4], chrome133a[5][6], chrome136[6] chrome132, chrome134, chrome135
Chrome Android chrome99_android, chrome131_android [4] chrome132_android, chrome133_android, chrome134_android, chrome135_android
Chrome iOS N/A coming soon
Safari [7] safari153 [2], safari155 [2], safari170 [1], safari180 [4], safari184 [6], safari260 [8] coming soon
Safari iOS [7] safari172_ios[1], safari180_ios[4], safari184_ios [6], safari260_ios [8] coming soon
Firefox firefox133[5], firefox135[7] coming soon
Firefox Android N/A firefox135_android
Tor tor145 [7] coming soon
Edge edge99, edge101 edge133, edge135
Opera N/A coming soon
Brave N/A coming soon

Notes:

  1. Added in version 0.6.0.
  2. Fixed in version 0.6.0, previous http2 fingerprints were not correct.
  3. Added in version 0.7.0.
  4. Added in version 0.8.0.
  5. Added in version 0.9.0.
  6. The version postfix -a(e.g. chrome133a) means that this is an alternative version, i.e. the fingerprint has not been officially updated by browser, but has been observed because of A/B testing.
  7. Added in version 0.10.0.
  8. Added in version 0.11.0.
  9. Since 0.11.0, the format safari184_ios is preferred over safari18_4_ios, both are supported, but the latter is quite confusing and hard to parse.
  10. Added in 0.12.0.

Asyncio

from curl_cffi import AsyncSession

async with AsyncSession() as s:
    r = await s.get("https://example.com")

More concurrency:

import asyncio
from curl_cffi import AsyncSession

urls = [
    "https://google.com/",
    "https://facebook.com/",
    "https://twitter.com/",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get(url)
        tasks.append(task)
    results = await asyncio.gather(*tasks)

For low-level APIs, Scrapy integration and other advanced topics, see the docs for more details.

WebSockets

from curl_cffi import WebSocket

def on_message(ws: WebSocket, message: str | bytes):
    print(message)

ws = WebSocket(on_message=on_message)
ws.run_forever("wss://api.gemini.com/v1/marketdata/BTCUSD")

Asyncio WebSockets

import asyncio
from curl_cffi import AsyncSession

async with AsyncSession() as s:
    ws = await s.ws_connect("wss://echo.websocket.org")
    await asyncio.gather(*[ws.send_str("Hello, World!") for _ in range(10)])
    async for message in ws:
        print(message)

Ecosystem

Acknowledgement

  • Originally forked from multippt/python_curl_cffi, which is under the MIT license.
  • Headers/Cookies files are copied from httpx, which is under the BSD license.
  • Asyncio support is inspired by Tornado's curl http client.
  • The synchronous WebSocket API is inspired by websocket_client.
  • The asynchronous WebSocket API is inspired by aiohttp.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

curl_cffi-0.12.0.tar.gz (150.5 kB view details)

Uploaded Source

Built Distributions

curl_cffi-0.12.0-cp39-abi3-win_amd64.whl (1.6 MB view details)

Uploaded CPython 3.9+Windows x86-64

curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_x86_64.whl (8.7 MB view details)

Uploaded CPython 3.9+musllinux: musl 1.1+ x86-64

curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_aarch64.whl (8.8 MB view details)

Uploaded CPython 3.9+musllinux: musl 1.1+ ARM64

curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (7.5 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ i686

curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.9 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

curl_cffi-0.12.0-cp39-abi3-macosx_11_0_arm64.whl (3.0 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

curl_cffi-0.12.0-cp39-abi3-macosx_10_9_x86_64.whl (5.7 MB view details)

Uploaded CPython 3.9+macOS 10.9+ x86-64

File details

Details for the file curl_cffi-0.12.0.tar.gz.

File metadata

  • Download URL: curl_cffi-0.12.0.tar.gz
  • Upload date:
  • Size: 150.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for curl_cffi-0.12.0.tar.gz
Algorithm Hash digest
SHA256 01770d120e2ab82ad076687c7038abe4d614c7e13d7a5378520b027df529dbe7
MD5 e4791c10712a592f9812c3fcb11aa529
BLAKE2b-256 f491feaf237ab7c5e06cc0a87ed7acdfbd4103beaa3ca5666d78e80b4a044e47

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: curl_cffi-0.12.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.8

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 cdcbc492a68b7f3592a4dc4eb742281aa74d220f55affbd84645795a7fdb3ddc
MD5 0b25fe17cefa71618a9402436f29c13a
BLAKE2b-256 226e0194d04312fbf6eed0d0fea6dfd361795fcfd53e9dca259a8ad45ff1ccca

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 b9650b85964ed06c634cfff4ce926255b80195f73edf629a1272b1a19908811d
MD5 04b434c890b15fedbefdae719fb914a9
BLAKE2b-256 aa012ff55d2d73b0d4605fff32112b33894e6b6d79106406d9e1185d680e8930

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 c0875e85eda5a5314bf1974ad1ecdcdd61811759b820b6617ec7be6daf85a1a3
MD5 975037d8b0bc90c547ad90eba57639a1
BLAKE2b-256 574f000397ecd769d663dbfdaea10173e4ee860379118c81e40d27faea4382c6

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 18aadb313a1fe23098e867f9e6e42c57c5c68985521a1fe5fb8ca15bb990341b
MD5 740ef5f5632073f3317eb5343191bc1d
BLAKE2b-256 fe0b559443dd7fcdeb424f353c239db3aa0421659f9bab26a89fc65b703cb486

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 1df994f9ccf27b391259ba0157d415b1e60f01e59914f75280cb9c1eceb3a3c8
MD5 fb3c920c880085f8772adcc042f95663
BLAKE2b-256 3ffd700edf216767c0e25632e24ec66dfb4a047cf6966d3246507f8a384970cd

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 368dd6e354933c62d3b35afbecdc5e7e7817ba748db0d23f7276f89b7eec49e8
MD5 9c9a101f2255feda94ff54e2a3aae7a0
BLAKE2b-256 138310addc9189c8eeb35e7b4c13033e84e174f534caef5df9f520403bcd546d

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8f7f1745700fcd4f6d5681cdca27426cc3005c3e17ca9a66fe5753c5001a7314
MD5 efa8844c8b650f6661b10d19adcde214
BLAKE2b-256 cd74e63d74451dcd0a793da1983024f36bbe400e8a3be8eaf860c004f66d489a

See more details on using hashes here.

File details

Details for the file curl_cffi-0.12.0-cp39-abi3-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.12.0-cp39-abi3-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 e663d6692aa923a60fd2f050dad4cccd317dc7dd3d9edceb5230f68017e811eb
MD5 da29d3405970d552d6d741f5a776021c
BLAKE2b-256 da58587f76b06fd7c4176b033275ab63b90e32b6904e5d7c7844311584376cdc

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page