libcurl ffi bindings for Python, with impersonation support
Project description
curl_cffi
Python binding for curl-impersonate via CFFI.
Unlike other pure python http clients like httpx
or requests
, this package can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Install
pip install --upgrade curl_cffi
This should work for Linux(x86_64/aarch64), macOS(Intel), Windows(amd64). If it does not
work, you may need to compile and install curl-impersonate
first.
Usage
requests/httpx
-like API:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
# proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
# sessions are supported
s = requests.Session()
s.get()
print(s.cookies)
Supported impersonate versions:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome101")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See example.py
or tests/
for more examples.
API
Requests: almost the same as requests.
Curl object:
setopt(CurlOpt, value)
: Sets curl options as incurl_easy_setopt
perform()
: Performs curl request, as incurl_easy_perform
getinfo(CurlInfo)
: Gets information in response after curl perform, as incurl_easy_getinfo
close()
: Closes and cleans up the curl object, as incurl_easy_cleanup
Enum values to be used with setopt
and getinfo
, and can be accessed from CurlOpt
and CurlInfo
.
Trouble Shooting
Pyinstaller ModuleNotFoundError: No module named '_cffi_backend'
You need to tell pyinstaller to pack cffi and data files inside the package:
pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi
Using https proxy, error: OPENSSL_internal:WRONG_VERSION_NUMBER
You are messing up https-over-http proxy and https-over-https proxy, for most cases, you
should change {"https": "https://localhost:3128"}
to {"https": "http://localhost:3128"}
.
Note the protocol in the url for https proxy is http
not https
.
See this issue for a detailed explaination.
Current Status
This implementation is very hacky now, but it works for most common systems.
When people installing other python curl bindings, like pycurl
, they often face
compiling issues or OpenSSL issues, so I really hope that this package can be distributed
as a compiled binary package, uses would be able to use it by a simple pip install
, no
more compile errors.
For now, I just download the pre-compiled libcurl-impersonate
from github and build a
bdist wheel, which is a binary package format used by PyPI, and upload it. However, the
right way is to download curl and curl-impersonate sources on our side and compile them
all together.
Help wanted!
TODOs:
- Write docs.
- Binary package for macOS(Intel) and Windows.
- Support musllinux(alpine) and macOS(Apple Silicon) bdist by building from source.
- Exclude the curl headers from source, download them when building.
- Update curl header files and constants via scripts.
- Implement
requests.Session/httpx.Client
. - Create ABI3 wheels to reduce package size and build time.
- Set default headers as in curl-impersonate wrapper scripts.
Change Log
- 0.3.0, copied more code from
httpx
to support session- Add
requests.Session
- Breaking change:
Response.cookies
changed fromhttp.cookies.SimpleCookie
tocurl_cffi.requests.Cookies
- Using ABI3 wheels to reduce package size.
- Add
Acknowledgement
- This package was originally forked from https://github.com/multippt/python_curl_cffi
- headers/cookies files are copied from https://github.com/encode/httpx/blob/master/httpx/_models.py, which is under BSD License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for curl_cffi-0.3.0-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 807bb5f9d00b9b83d68ea0226ee3cfa92408340bed4ece4c60d9866cd688ff9f |
|
MD5 | 59df40ff563718e66b4e6afb23f71533 |
|
BLAKE2b-256 | 5e721955d4f7a9d5a441e4485e9520be9daec3e739a808296b2852e10c428a10 |
Hashes for curl_cffi-0.3.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d37ff746e0bb8be0ace528c7952d566105e2070c7a1f8066b9db601922df825 |
|
MD5 | 5799239b240a3ea65cbe0c447a11cfb0 |
|
BLAKE2b-256 | 3185a24813c75c1bf5c6038f8640bef821601ef59c38e1edf991f792172d695d |
Hashes for curl_cffi-0.3.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d3fdfaaa03d3f7e3d16d905d79de583da86137af26a82875938a99cd1b88a7c |
|
MD5 | 903ce41362d82f7d6711726dcd9dcc71 |
|
BLAKE2b-256 | d37a7c7dbdbcaa13da24ca2e76cbf659a9be866c7f42edec2988534171251b3a |
Hashes for curl_cffi-0.3.0-cp37-abi3-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1af97a4b126a9863d18807a4b2223a7df1ecd354efd2af972888e1f09d19992c |
|
MD5 | a402b3c40f0b381db9f05047ce5d65ce |
|
BLAKE2b-256 | 051b1f770f2e2cd3a10990bac7cd496703726a72ded130f5e73be6fc0206a575 |