Pre-release
This release is a pre-release and may not be stable for production use.
curl_cffi
Python binding for curl-impersonate via cffi.
Documentation | 中文 README | Discuss on Telegram
Unlike other pure python http clients like httpx or requests, curl_cffi can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asynciowith proxy rotation on each request. - Supports http 2.0, which requests does not.
- Supports websocket.
| library | requests | aiohttp | httpx | pycurl | curl_cffi |
|---|---|---|---|---|---|
| http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
| sync | ✅ | ❌ | ✅ | ✅ | ✅ |
| async | ❌ | ✅ | ✅ | ❌ | ✅ |
| websocket | ❌ | ✅ | ❌ | ❌ | ✅ |
| fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
| speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH.
To install beta releases:
pip install curl_cffi --upgrade --pre
To install unstable version from GitHub:
git clone https://github.com/yifeikong/curl_cffi/
cd curl_cffi
make preprocess
pip install .
Usage
Use the latest impersonate versions, do NOT copy chrome110 here without changing.
requests-like
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by my fork of curl-impersonate:
However, only Chrome-like browsers are supported. Firefox support is tracked in #59.
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome116 [1]
- chrome119 [1]
- chrome120 [1]
- chrome99_android
- edge99
- edge101
- safari15_3 [2]
- safari15_5 [2]
- safari17_0 [1]
- safari17_2_ios [1]
Notes:
- Added in version
0.6.0. - fixed in version
0.6.0, previous http2 fingerprints were not correct.
asyncio
from curl_cffi.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from curl_cffi.requests import AsyncSession
urls = [
"https://google.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get(url)
tasks.append(task)
results = await asyncio.gather(*tasks)
WebSockets
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
curl-like
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
scrapy
If you are using scrapy, check out these middlewares:
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The WebSocket API is inspired by websocket_client.
[Sponsor] Bypass Cloudflare with API
Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance). Click here to register: https://yescaptcha.com/i/stfnIO
Sponsor
1 maintainer
Release files for curl-cffi 0.6.0b9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| curl_cffi-0.6.0b9.tar.gz | 45.3 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| curl_cffi-0.6.0b9-cp38-abi3-win_amd64.whl | CPython 3.8 | abi3 | Windows x86-64 | Details |
| curl_cffi-0.6.0b9-cp38-abi3-win32.whl | CPython 3.8 | abi3 | Windows x86-32 | Details |
| curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.8 | abi3 | Linux glibc 2.17+ ARM64 | Details |
| curl_cffi-0.6.0b9-cp38-abi3-macosx_11_0_arm64.whl | CPython 3.8 | abi3 | macOS 11.0+ ARM64 | Details |
| curl_cffi-0.6.0b9-cp38-abi3-macosx_10_9_x86_64.whl | CPython 3.8 | abi3 | macOS 10.9+ x86-64 | Details |
Total release size: 25.3 MB
Release files / curl_cffi-0.6.0b9.tar.gz
| Download URL | curl_cffi-0.6.0b9.tar.gz |
|---|---|
| Size | 45.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7d9a6a68c2aef90359a8d012edf7bdd81eaa1b16af684f21febf00fb54dd8d73
|
|
BLAKE2b-256 checksum How to use checksums |
40a6ba96529f152164da561267a7793c691b09a5e71d0a2a9ab3cb2a17aba0f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-win_amd64.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-win_amd64.whl |
|---|---|
| Size | 2.4 MB |
| Tags | CPython 3.8 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
00b538b67c60cd1c3286c7319813a3564079c62ccf126cc1462fff428f0c3be5
|
|
BLAKE2b-256 checksum How to use checksums |
e9dac549c73decd1aeb7b6f33cea579ec264dcc749e5658ce5c9b3cd93ce6cd9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-win32.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-win32.whl |
|---|---|
| Size | 2.4 MB |
| Tags | CPython 3.8 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
92baeb4544a2353eb882a2d827ba546f7430655d04272b0678eedf2ae4618309
|
|
BLAKE2b-256 checksum How to use checksums |
4b892ee3528deddf2fd5ce4212b401191851a1cb990d9f05fe8fc5cccad971f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
c02e53bf3774a02b20c5a4458af965555d11755119e575d91c2984cedd1baa9c
|
|
BLAKE2b-256 checksum How to use checksums |
1a1e5f4e5dcb9102bbe9c3a3271fa09ce8825920d322772980cf6a9fa9f66a3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 6.6 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
1e18bb9f73a31f7f6c61b7341c1a4742b200779c177fe06276a3a03fdb270b13
|
|
BLAKE2b-256 checksum How to use checksums |
25c72f9a4373a4a0125f27b3e39d136d77c12efe879302233f9dd50d22890d57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-macosx_11_0_arm64.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 2.3 MB |
| Tags | CPython 3.8 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
984ac36e20070206e8729abdb00675dae233ea53fffe678c33b3869b26a9c537
|
|
BLAKE2b-256 checksum How to use checksums |
e67d27cd8c6c8b8591f9b14f8adc7bb209753069028cd69328540c6fdd648594
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / curl_cffi-0.6.0b9-cp38-abi3-macosx_10_9_x86_64.whl
| Download URL | curl_cffi-0.6.0b9-cp38-abi3-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.7 MB |
| Tags | CPython 3.8 abi3 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
a0385edcbb545e58757849308a47292a895ab3b3e36f5f1c2e19e8d8db546022
|
|
BLAKE2b-256 checksum How to use checksums |
2d4bab92fb76ce785e4893cfb5078014da701bc7204a1aa577d8494835835fbd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
