libcurl ffi bindings for Python, with impersonation support
Project description
curl_cffi
Python binding for curl-impersonate via CFFI.
Unlike other pure python http clients like httpx
or requests
, this package can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Install
pip install --upgrade curl_cffi
This should work for Linux(x86_64/aarch64), macOS(Intel/Apple Silicon), Windows(amd64).
If it does not work, you may need to compile and install curl-impersonate
first.
Usage
requests/httpx
-like API:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
# proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
# socks proxies are also supported
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome101")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See example.py
or tests/
for more examples.
API
Requests: almost the same as requests.
Curl object:
setopt(CurlOpt, value)
: Sets curl options as incurl_easy_setopt
perform()
: Performs curl request, as incurl_easy_perform
getinfo(CurlInfo)
: Gets information in response after curl perform, as incurl_easy_getinfo
close()
: Closes and cleans up the curl object, as incurl_easy_cleanup
Enum values to be used with setopt
and getinfo
, and can be accessed from CurlOpt
and CurlInfo
.
Trouble Shooting
Pyinstaller ModuleNotFoundError: No module named '_cffi_backend'
You need to tell pyinstaller to pack cffi and data files inside the package:
pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi
Using https proxy, error: OPENSSL_internal:WRONG_VERSION_NUMBER
You are messing up https-over-http proxy and https-over-https proxy, for most cases, you
should change {"https": "https://localhost:3128"}
to {"https": "http://localhost:3128"}
.
Note the protocol in the url for https proxy is http
not https
.
See this issue for a detailed explaination.
Current Status
This implementation is very hacky now, but it works for most common systems.
When people installing other python curl bindings, like pycurl
, they often face
compiling issues or OpenSSL issues, so I really hope that this package can be distributed
as a compiled binary package, uses would be able to use it by a simple pip install
, no
more compile errors.
For now, I just download the pre-compiled libcurl-impersonate
from github and build a
bdist wheel, which is a binary package format used by PyPI, and upload it. However, the
right way is to download curl and curl-impersonate sources on our side and compile them
all together.
Help wanted!
TODOs:
- Write docs.
- Binary package for macOS(Intel/AppleSilicon) and Windows.
- Support musllinux(alpine) bdist by building from source.
- Exclude the curl headers from source, download them when building.
- Update curl header files and constants via scripts.
- Implement
requests.Session/httpx.Client
. - Create ABI3 wheels to reduce package size and build time.
- Set default headers as in curl-impersonate wrapper scripts.
Change Log
- 0.3.0, copied more code from
httpx
to support session- Add
requests.Session
- Breaking change:
Response.cookies
changed fromhttp.cookies.SimpleCookie
tocurl_cffi.requests.Cookies
- Using ABI3 wheels to reduce package size.
- Add
Acknowledgement
- This package was originally forked from https://github.com/multippt/python_curl_cffi
- headers/cookies files are copied from https://github.com/encode/httpx/blob/master/httpx/_models.py, which is under BSD License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for curl_cffi-0.3.2-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e02cbe1668702b95a4a17b7bee27f57a8594856284f644481d03f542b7505ad |
|
MD5 | 0666e3beec9dfa9e83376ad1ce8eb776 |
|
BLAKE2b-256 | 1cf621033c0093278c1e886e340c8d6867064767a459301b1ace2f70ae5af60d |
Hashes for curl_cffi-0.3.2-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ae50a2c8dd9f829db9c3d8110d442e8fafc4fb8675bb2c9c1f5fba34654160e |
|
MD5 | 19973a6ee7f616922a209c74b39b0503 |
|
BLAKE2b-256 | e860abdeb3dec907f2c45820f84afa16717d1933658280a07d967739b59d21fe |
Hashes for curl_cffi-0.3.2-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fa335b7002974eda0341d344a7532fcbf0fd77fa3df292a2c2f6167fd1005ba |
|
MD5 | 31b994493f17f8506ccab80f8b421789 |
|
BLAKE2b-256 | 88c186fa3d8f82bf77b473ef04e12cb9163de308cbdf3215d3cb4d761038dd44 |
Hashes for curl_cffi-0.3.2-cp37-abi3-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 627ecc8eb60a60f0bfb54f33728616dad4ae6fa60f07bd478a5dd3f8ab451ff4 |
|
MD5 | 69515d0d3767e9c237eea837c3c396f8 |
|
BLAKE2b-256 | 8387d6ce016c1531c7c2ea5df68102e101d7d91abb3e7f451e8e3af42a72cea8 |