libcurl ffi bindings for Python, with impersonation support
Project description
curl_cffi
Python binding for curl-impersonate via CFFI.
Unlike other pure python http clients like httpx
or requests
, this package can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Install
pip install --upgrade curl_cffi
This should work for Linux(x86_64/aarch64), macOS(Intel/Apple Silicon), Windows(amd64).
If it does not work, you may need to compile and install curl-impersonate
first.
Usage
requests/httpx
-like API:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
# proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
# socks proxies are also supported
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome101")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See example.py
or tests/
for more examples.
API
Requests: almost the same as requests.
Curl object:
setopt(CurlOpt, value)
: Sets curl options as incurl_easy_setopt
perform()
: Performs curl request, as incurl_easy_perform
getinfo(CurlInfo)
: Gets information in response after curl perform, as incurl_easy_getinfo
close()
: Closes and cleans up the curl object, as incurl_easy_cleanup
Enum values to be used with setopt
and getinfo
, and can be accessed from CurlOpt
and CurlInfo
.
Trouble Shooting
Pyinstaller ModuleNotFoundError: No module named '_cffi_backend'
You need to tell pyinstaller to pack cffi and data files inside the package:
pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi
Using https proxy, error: OPENSSL_internal:WRONG_VERSION_NUMBER
You are messing up https-over-http proxy and https-over-https proxy, for most cases, you
should change {"https": "https://localhost:3128"}
to {"https": "http://localhost:3128"}
.
Note the protocol in the url for https proxy is http
not https
.
See this issue for a detailed explaination.
Current Status
This implementation is very hacky now, but it works for most common systems.
When people installing other python curl bindings, like pycurl
, they often face
compiling issues or OpenSSL issues, so I really hope that this package can be distributed
as a compiled binary package, uses would be able to use it by a simple pip install
, no
more compile errors.
For now, I just download the pre-compiled libcurl-impersonate
from github and build a
bdist wheel, which is a binary package format used by PyPI, and upload it. However, the
right way is to download curl and curl-impersonate sources on our side and compile them
all together.
Help wanted!
TODOs:
- Write docs.
- Binary package for macOS(Intel/AppleSilicon) and Windows.
- Support musllinux(alpine) bdist by building from source.
- Exclude the curl headers from source, download them when building.
- Update curl header files and constants via scripts.
- Implement
requests.Session/httpx.Client
. - Create ABI3 wheels to reduce package size and build time.
- Set default headers as in curl-impersonate wrapper scripts.
Change Log
-
0.3.6
- Updated to curl-impersonate v0.5.4, supported chrome107 and chrome110
-
0.3.0, copied more code from
httpx
to support session- Add
requests.Session
- Breaking change:
Response.cookies
changed fromhttp.cookies.SimpleCookie
tocurl_cffi.requests.Cookies
- Using ABI3 wheels to reduce package size.
- Add
Acknowledgement
- This package was originally forked from https://github.com/multippt/python_curl_cffi
- headers/cookies files are copied from https://github.com/encode/httpx/blob/master/httpx/_models.py, which is under BSD License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for curl_cffi-0.3.7-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0762c7834aaf7187f4edf0a16bb4c748bf0762f3d15c99731becdbb36a03e370 |
|
MD5 | a929b90dc71fc497a62cd60fa8bafd4c |
|
BLAKE2b-256 | 73e7c8993587234c804f0005d481717c5819a38dd796b0de076ddbc0c9092329 |
Hashes for curl_cffi-0.3.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ae929e7a5cdde019e9c2889dec6094b0df565e75c9bf9efd587b3ba5501cebd |
|
MD5 | 307b5e999f2e4dbeaf8c4fac8ad9dd23 |
|
BLAKE2b-256 | 48c3d5501d6c04a31f6691a6c9bc27112514cf9514081afc89f85db70167f3e3 |
Hashes for curl_cffi-0.3.7-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bfd37e02d2f7ccc2227dcc771835994563672418f7ab68d7e5e7b7932a773f6 |
|
MD5 | 7300c3cadba2440d835c484bbb605811 |
|
BLAKE2b-256 | ca65636c94eff7f3aa6cc95516a54a6bceae5684470b9cdb23d16c779b86d0e2 |
Hashes for curl_cffi-0.3.7-cp37-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9232ed307c6d9fde6a8206a0449901b0921f329c687b2f295ddfd4d1e67cfc15 |
|
MD5 | cd4cfda3fcdfb8b6fac0f8886a2c6615 |
|
BLAKE2b-256 | 667c3136d835e4673b23ad5e01e8ee5ce9b63e84e5de7668a8a298517645552b |
Hashes for curl_cffi-0.3.7-cp37-abi3-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 916fddb312dd6fca1ece55a3331f30985af58d7a101b7481141d928db7565a0f |
|
MD5 | a6928613f869868a6e24330da4f16f1a |
|
BLAKE2b-256 | f7ee390dbf7279c9eb75f5204123c491f49b0d198045d2ffe3b9e54095c90ade |