libcurl ffi bindings for Python, with impersonation support
Project description
curl_cffi
Python binding for curl-impersonate via cffi.
Unlike other pure python http clients like httpx
or requests
, curl_cffi
can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asyncio
with proxy rotation on each request. - Supports http 2.0, which requests does not.
library | requests | aiohttp | httpx | pycurl | curl_cffi |
---|---|---|---|---|---|
http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
sync | ✅ | ❌ | ✅ | ✅ | ✅ |
async | ❌ | ✅ | ✅ | ❌ | ✅ |
fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH
.
To install beta releases:
pip install curl_cffi --pre
Usage
requests-like
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by curl-impersonate:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
asyncio
from curl_cffi.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from curl_cffi.requests import AsyncSession
urls = [
"https://googel.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get("https://example.com")
tasks.append(task)
results = await asyncio.gather(*tasks)
curl-like
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
If you are using scrapy, check out this middleware: tieyongjie/scrapy-fingerprint
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file curl_cffi-0.5.10.tar.gz
.
File metadata
- Download URL: curl_cffi-0.5.10.tar.gz
- Upload date:
- Size: 35.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
55bac4b73e2d80ceeaabea33270fc8ca6ace594128a46710242f2e688b4f8bfc
|
|
MD5 |
e9324563a70356c82a766324e927a003
|
|
BLAKE2b-256 |
505743d0e4475c336282deb7acf5285c35fbdce6a041d1583dbe30e0d761117d
|
File details
Details for the file curl_cffi-0.5.10-cp37-abi3-win_amd64.whl
.
File metadata
- Download URL: curl_cffi-0.5.10-cp37-abi3-win_amd64.whl
- Upload date:
- Size: 2.6 MB
- Tags: CPython 3.7+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
f9a1874b860c4e8db49bdfd9b9d4dc39999a1397d271ec78624c35c838e9e92a
|
|
MD5 |
341868ec10c7892bf44ce6275b1374cd
|
|
BLAKE2b-256 |
4e700d4d667287d54cb4daf8ce67068565c8e09e4f839a835119cd27f229a446
|
File details
Details for the file curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 7.2 MB
- Tags: CPython 3.7+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
1b2bc8822d23415f6533c8b750475e9bbc76764025fe1dcb5866dc033607fd7b
|
|
MD5 |
3f3678fadcdad60d9324b728047275ad
|
|
BLAKE2b-256 |
93040d57a51690451437f72563031036ea4f4b529bc0cfd8e8d93487075cbe80
|
File details
Details for the file curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 6.9 MB
- Tags: CPython 3.7+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
b537595b9610a4dd0927c09823925b4e32b1ce0fd04385bfc5bb72ab830720e6
|
|
MD5 |
0982e65b8d4553488596fc9b79dc88b6
|
|
BLAKE2b-256 |
162edf2c475bae057af3fff4d288a388062032e7005ad312e8f55490faef82e2
|
File details
Details for the file curl_cffi-0.5.10-cp37-abi3-macosx_11_0_arm64.whl
.
File metadata
- Download URL: curl_cffi-0.5.10-cp37-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 2.2 MB
- Tags: CPython 3.7+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.16
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
9937b8e13b1a6963c63e155b6621ec74649965105efedb919bc226fe731861cc
|
|
MD5 |
a2c356014da1d4b9ec0053f6729f1efe
|
|
BLAKE2b-256 |
23253c9fdb7e08e4722eb9e7ebcc59c74d7eaf32ca4a878a3fdf581030940431
|
File details
Details for the file curl_cffi-0.5.10-cp37-abi3-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: curl_cffi-0.5.10-cp37-abi3-macosx_10_9_x86_64.whl
- Upload date:
- Size: 4.9 MB
- Tags: CPython 3.7+, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
892603dab5e56fb72bfff7ae969136138971f63f63defe98232e1ec55cb0f1c6
|
|
MD5 |
cb3877f3efbf7c389781131af43a356d
|
|
BLAKE2b-256 |
301872d6d4f5ec0db3ea68dde9ad775f9985ce22e699a47ab5b4cfcf8d74f3f5
|