libcurl ffi bindings for Python, with impersonation support
Project description
curl_cffi
Python binding for curl-impersonate via cffi.
Documentation | 中文 README | Discuss on Telegram
Unlike other pure python http clients like httpx
or requests
, curl_cffi
can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asyncio
with proxy rotation on each request. - Supports http 2.0, which requests does not.
- Supports websocket.
library | requests | aiohttp | httpx | pycurl | curl_cffi |
---|---|---|---|---|---|
http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
sync | ✅ | ❌ | ✅ | ✅ | ✅ |
async | ❌ | ✅ | ✅ | ❌ | ✅ |
websocket | ❌ | ✅ | ❌ | ❌ | ✅ |
fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH
.
To install beta releases:
pip install curl_cffi --pre
Usage
Use the latest impersonate versions, do NOT copy chrome110
here without changing.
requests-like
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by my fork of curl-impersonate:
However, only Chrome-like browsers are supported. Firefox support is tracked in #55
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome116
- chrome119
- chrome120
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
- safari17_2_ios
asyncio
from curl_cffi.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from curl_cffi.requests import AsyncSession
urls = [
"https://googel.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get("https://example.com")
tasks.append(task)
results = await asyncio.gather(*tasks)
WebSockets
from curl_cffi.requests import Session, WebSocket
def on_message(ws: WebSocket, message):
print(message)
with Session() as s:
ws = s.ws_connect(
"wss://api.gemini.com/v1/marketdata/BTCUSD",
on_message=on_message,
)
ws.run_forever()
curl-like
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
If you are using scrapy, check out these middlewares:
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The WebSocket API is inspired by websocket_client
Sponsor
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for curl_cffi-0.6.0b7-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 271892078905981880a40fbdf2d53c0e5814dcc102836d69d30772b731898de4 |
|
MD5 | 45de7f1da8aa00984b2293d4099618b9 |
|
BLAKE2b-256 | 127fc289f4c71ff4249799ba87491f796a661f2760bd089299192a245a53d3ca |
Hashes for curl_cffi-0.6.0b7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4df7a48a313688cb0dc2924229a3604b71700494a31ac7920701a2f930b49cdf |
|
MD5 | 8a239a372d0f8fb019efe40f83e86f65 |
|
BLAKE2b-256 | 54b6e0ed7de404035bf35bd89ce73b97d0ffc7eece855959b8abb19ad483ffb9 |
Hashes for curl_cffi-0.6.0b7-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef7e73e3148b482f69e30c7651dde5fd98900df597c0cdfdf1f433f3fb819037 |
|
MD5 | 4b22ce05895ddfdbba8921ea95738080 |
|
BLAKE2b-256 | deb573fd209749d7ae3eeb0dec337f2688f6f461133d11b81051007553f9fb63 |
Hashes for curl_cffi-0.6.0b7-cp37-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dcb245dacaec556ed48dc8e4c609bdf5dfe52b84073a124ee3f5e9342e5a7af |
|
MD5 | 3d67bd872085399c980671fc24b1057c |
|
BLAKE2b-256 | b5374969c25cd9e84ac339400dcb07b1b20fa4f345461ffe4349f452b1eb4a9f |
Hashes for curl_cffi-0.6.0b7-cp37-abi3-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0939e97e8f999bc51bde6a70e19b23aaee542ccc341955989d81e5b11d616cef |
|
MD5 | 288240ad44f533b08e31f56d1509de16 |
|
BLAKE2b-256 | 6db5b13116d7ae2fb0b09c8d9527ca5ea48b6c37be886f8fdf8349d06dda0228 |