curl_cffi
Python binding for curl-impersonate via cffi.
Unlike other pure python http clients like httpx or requests, curl_cffi can
impersonate browsers' TLS/JA3 and HTTP/2 fingerprints. If you are blocked by some
website for no obvious reason, you can give curl_cffi a try.
Scrapfly is an enterprise-grade solution providing Web Scraping API that aims to simplify the scraping process by managing everything: real browser rendering, rotating proxies, and fingerprints (TLS, HTTP, browser) to bypass all major anti-bots. Scrapfly also unlocks the observability by providing an analytical dashboard and measuring the success rate/block rate in detail.
Scrapfly is a good solution if you are looking for a cloud-managed solution for curl_cffi.
If you are managing TLS/HTTP fingerprint by yourself with curl_cffi, they also maintain
this tool to convert curl
command into python curl_cffi code!
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asynciowith proxy rotation on each request. - Supports http 2.0, which requests does not.
- Supports websocket.
| requests | aiohttp | httpx | pycurl | curl_cffi | |
|---|---|---|---|---|---|
| http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
| sync | ✅ | ❌ | ✅ | ✅ | ✅ |
| async | ❌ | ✅ | ✅ | ❌ | ✅ |
| websocket | ❌ | ✅ | ❌ | ❌ | ✅ |
| fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
| speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux, macOS and Windows out of the box.
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH.
To install beta releases:
pip install curl_cffi --upgrade --pre
To install unstable version from GitHub:
git clone https://github.com/yifeikong/curl_cffi/
cd curl_cffi
make preprocess
pip install .
Usage
curl_cffi comes with a low-level curl API and a high-level requests-like API.
Use the latest impersonate versions, do NOT copy chrome110 here without changing.
requests-like
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# To keep using the latest browser version as `curl_cffi` updates,
# simply set impersonate="chrome" without specifying a version.
# Other similar values are: "safari" and "safari_ios"
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome")
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome110", proxies=proxies)
Sessions
s = requests.Session()
# httpbin is a http test website, this endpoint makes the server set cookies
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
# retrieve cookies again to verify
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by my fork of curl-impersonate:
However, only Chrome-like browsers are supported. Firefox support is tracked in #59.
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome116 [1]
- chrome119 [1]
- chrome120 [1]
- chrome99_android
- edge99
- edge101
- safari15_3 [2]
- safari15_5 [2]
- safari17_0 [1]
- safari17_2_ios [1]
Notes:
- Added in version
0.6.0. - Fixed in version
0.6.0, previous http2 fingerprints were not correct.
asyncio
from curl_cffi.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from curl_cffi.requests import AsyncSession
urls = [
"https://google.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get(url)
tasks.append(task)
results = await asyncio.gather(*tasks)
WebSockets
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
For low-level APIs, Scrapy integration and other advanced topics, see the docs for more details.
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The WebSocket API is inspired by websocket_client.
[Sponsor] Bypass Cloudflare with API
Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance). Click here to register: https://yescaptcha.com/i/stfnIO
[Sponsor] ScrapeNinja
ScrapeNinja is a web scraping API with two engines: fast, with high performance and TLS fingerprint; and slower with a real browser under the hood.
ScrapeNinja handles headless browsers, proxies, timeouts, retries, and helps with data extraction, so you can just get the data in JSON. Rotating proxies are available out of the box on all subscription plans.
Sponsor
1 maintainer
Release files for curl-cffi 0.6.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| curl_cffi-0.6.3.tar.gz | 125.2 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| curl_cffi-0.6.3-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| curl_cffi-0.6.3-cp38-abi3-win32.whl | CPython 3.8 | abi3 | Windows x86-32 | Details |
| curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_x86_64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ x86-64 | Details |
| curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_aarch64.whl | CPython 3.8 | abi3 | Linux musl 1.1+ ARM64 | Details |
| curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-32 | Details |
| curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ ARM64 | Details |
| curl_cffi-0.6.3-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| curl_cffi-0.6.3-cp38-abi3-macosx_10_9_x86_64.whl | CPython 3.8 | abi3 | macOS 10.9+ x86-64 | Details |
Total release size: 38.4 MB
Release files / curl_cffi-0.6.3.tar.gz
| Download URL | curl_cffi-0.6.3.tar.gz |
|---|---|
| Size | 125.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0d2d07467590d66982f29a8dc9050b6a1f41a6c4bb44dcbf24108b13cf71a797
|
|
BLAKE2b-256 checksum How to use checksums |
09af32674f81e0865f7f11347e5e6acca78fd87790a2efde942fd7d25acdaa6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-win_amd64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
dd36513cd46eb8f2751d45e3aa47a989421d3f3a2bcc54abab487ae38549d91b
|
|
BLAKE2b-256 checksum How to use checksums |
7256808cde7c3fc833c5cf94e0218e25278edc77b9146f16c7176eeea120d17e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-win32.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-win32.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.8 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
0f37e3d761e37173462dbc95ae4216165dacef21f0d9ab6bb6802003516a156f
|
|
BLAKE2b-256 checksum How to use checksums |
0d9ae2738d39401a3de5cb62e0d223b4a042ab81a349d8c688c42173179ce3e5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_x86_64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_x86_64.whl |
|---|---|
| Size | 5.4 MB |
| Tags | CPython 3.8 Linux musl 1.1+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ea9b04df071368d13b02c51e87e3e51b46b924be61681fc280cb9633b42dac2c
|
|
BLAKE2b-256 checksum How to use checksums |
3c39c97eb0937ab51530a750d7bdf58af54750b5f36d9fff2008d000cec4c294
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_aarch64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-musllinux_1_1_aarch64.whl |
|---|---|
| Size | 5.7 MB |
| Tags | CPython 3.8 Linux musl 1.1+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
5d659acbe051805b9f9c82a7725d534b5842c1a9291159e7733e7c92782ef80b
|
|
BLAKE2b-256 checksum How to use checksums |
0376bdee80008359b37c684b74dcacd7b13e97db15a8663e6879f0b0a64e6b4d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.5 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
4b4712bc9a8a0d933aef7051eaa5448b9a1c662f25dd83967dbdd46aa3e418c0
|
|
BLAKE2b-256 checksum How to use checksums |
a29426f49de4ddc1085c71acefa9fbc8ecabc5b949055ab53217b8400a95012c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
5423c2f53f8889bcc9ce42455fc1c0c9487b480944f66aaa6ed5ce81e0fc5540
|
|
BLAKE2b-256 checksum How to use checksums |
27eb92d1ed7de100ee6aa361d8b50384d814c7fa1ba6f0d30aaf36b45c85663e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 5.2 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
760b7b837c86626f2c9518c3bad42aad3b6ccb455499b648bc98e8dee9f73891
|
|
BLAKE2b-256 checksum How to use checksums |
0f869a507111ae9d520126d48745cb44cbed66b509576d01a957d9cdab7fb4bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
d78a609c3b984df9a14022c0b5fc59c1c643c39fc4cb9100e110f7551339a194
|
|
BLAKE2b-256 checksum How to use checksums |
720ca1d55e9c4a72b06831977b8b133e619aaeefd07f650aabbc68acd2937ecd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
Release files / curl_cffi-0.6.3-cp38-abi3-macosx_10_9_x86_64.whl
| Download URL | curl_cffi-0.6.3-cp38-abi3-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.7 MB |
| Tags | CPython 3.8 abi3 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
ac60ac08f07640bd9ee6ee44310748931a3d49a7c8e878745f1817b46ff0719d
|
|
BLAKE2b-256 checksum How to use checksums |
b2091dae204b163997a1a800a2ab2f377a63e10a05ec1c1a5c60dc8818964283
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.3
|
