curl_cffi
Python binding for curl-impersonate via CFFI.
Unlike other pure python http clients like httpx or requests, this package can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Note on Chrome 110+ JA3 fingerprints
Chrome introduces ClientHello permutation in version 110, which means the order of
extensions will be random, thus JA3 fingerprints will be random. So, when comparing
JA3 fingerprints of curl_cffi and a browser, they may differ. However, this does not
mean that TLS fingerprints will not be a problem, ClientHello extension order is just
one factor of how servers can tell automated requests from browsers.
See more from this article and curl-impersonate notes
Install
pip install --upgrade curl_cffi
This should work for Linux(x86_64/aarch64), macOS(Intel/Apple Silicon), Windows(amd64).
If it does not work, you may need to compile and install curl-impersonate first.
Usage
requests/httpx-like API:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
# proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
# socks proxies are also supported
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
Alternatively, you can use the low-level curl-like API:
from curl_cffi import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)
c.impersonate("chrome101")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See example.py or tests/ for more examples.
API
Requests: almost the same as requests.
Curl object:
setopt(CurlOpt, value): Sets curl options as incurl_easy_setoptperform(): Performs curl request, as incurl_easy_performgetinfo(CurlInfo): Gets information in response after curl perform, as incurl_easy_getinfoclose(): Closes and cleans up the curl object, as incurl_easy_cleanup
Enum values to be used with setopt and getinfo, and can be accessed from CurlOpt and CurlInfo.
Trouble Shooting
Pyinstaller ModuleNotFoundError: No module named '_cffi_backend'
You need to tell pyinstaller to pack cffi and data files inside the package:
pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi
Using https proxy, error: OPENSSL_internal:WRONG_VERSION_NUMBER
You are messing up https-over-http proxy and https-over-https proxy, for most cases, you
should change {"https": "https://localhost:3128"} to {"https": "http://localhost:3128"}.
Note the protocol in the url for https proxy is http not https.
See this issue for a detailed explaination.
Current Status
This implementation is very hacky now, but it works for most common systems.
When people installing other python curl bindings, like pycurl, they often face
compiling issues or OpenSSL issues, so I really hope that this package can be distributed
as a compiled binary package, uses would be able to use it by a simple pip install, no
more compile errors.
For now, I just download the pre-compiled libcurl-impersonate from github and build a
bdist wheel, which is a binary package format used by PyPI, and upload it. However, the
right way is to download curl and curl-impersonate sources on our side and compile them
all together.
Help wanted!
TODOs:
- Write docs.
- Binary package for macOS(Intel/AppleSilicon) and Windows.
- Support musllinux(alpine) bdist by building from source.
- Exclude the curl headers from source, download them when building.
- Update curl header files and constants via scripts.
- Implement
requests.Session/httpx.Client. - Create ABI3 wheels to reduce package size and build time.
- Set default headers as in curl-impersonate wrapper scripts.
- Support stream in asyncio mode
Change Log
-
0.5.0
- Added asyncio support
-
0.4.0
- Removed c shim callback function, use cffi native callback function
-
0.3.6
- Updated to curl-impersonate v0.5.4, supported chrome107 and chrome110
-
0.3.0, copied more code from
httpxto support session- Add
requests.Session - Breaking change:
Response.cookieschanged fromhttp.cookies.SimpleCookietocurl_cffi.requests.Cookies - Using ABI3 wheels to reduce package size.
- Add
Acknowledgement
- This package was originally forked from https://github.com/multippt/python_curl_cffi , which is under the MIT license.
- headers/cookies files are copied from https://github.com/encode/httpx/blob/master/httpx/_models.py , which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
1 maintainer
Release files for curl-cffi 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| curl_cffi-0.5.0.tar.gz | 26.9 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| curl_cffi-0.5.0-cp37-abi3-win_amd64.whl | CPython 3.7 | abi3 | Windows x86-64 | Details |
| curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.7 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.7 | abi3 | Linux glibc 2.17+ ARM64 | Details |
| curl_cffi-0.5.0-cp37-abi3-macosx_11_0_arm64.whl | CPython 3.7 | abi3 | macOS 11.0+ ARM64 | Details |
| curl_cffi-0.5.0-cp37-abi3-macosx_10_9_x86_64.whl | CPython 3.7 | abi3 | macOS 10.9+ x86-64 | Details |
Total release size: 23.2 MB
Release files / curl_cffi-0.5.0.tar.gz
| Download URL | curl_cffi-0.5.0.tar.gz |
|---|---|
| Size | 26.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4086c2ce666e0975fd3aebb08f776ad5a73a283f15209236e8ba90f4c86b9137
|
|
BLAKE2b-256 checksum How to use checksums |
f3e370977a1d1d92f89c04a605422850be769e972e5b9f6ddf36b31b48099db1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.16
|
Release files / curl_cffi-0.5.0-cp37-abi3-win_amd64.whl
| Download URL | curl_cffi-0.5.0-cp37-abi3-win_amd64.whl |
|---|---|
| Size | 2.6 MB |
| Tags | CPython 3.7 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ce10b064b74ec44c118929d5b46f3ab8430ec2b81ec8099beaa6afa941e27566
|
|
BLAKE2b-256 checksum How to use checksums |
609e38f022006dcbf576d1c4453f066135a163c62c9f9cbbee44d9959492d837
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.16
|
Release files / curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 7.2 MB |
| Tags | CPython 3.7 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
3077340f7770e513b29fe14f488d21f2c7f2b79be1a7479c483c8c6b2541629a
|
|
BLAKE2b-256 checksum How to use checksums |
71e2e0c6d5ae58d5ea391bd0b1e1c81175cbea8b3cd8c02f366f2dc8e7d81197
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.16
|
Release files / curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | curl_cffi-0.5.0-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.7 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
30e98af21d771e4e2be3d0f90fe8bfccf81822b58011ae5ed797e098f0ebcbd0
|
|
BLAKE2b-256 checksum How to use checksums |
aa010320b335e9ed2414ade18045eb3fb88d4c6cbb56617ede52cbb3fd994544
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.16
|
Release files / curl_cffi-0.5.0-cp37-abi3-macosx_11_0_arm64.whl
| Download URL | curl_cffi-0.5.0-cp37-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.7 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
625597549d43d5cf6003446b01fb8162f752326b620473567b53704549b1b9af
|
|
BLAKE2b-256 checksum How to use checksums |
e97a7fa4b8dee4e9dfa8a1c68f8bb26f4ddcbd6f05d57ddda267557266fe839d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.8.16
|
Release files / curl_cffi-0.5.0-cp37-abi3-macosx_10_9_x86_64.whl
| Download URL | curl_cffi-0.5.0-cp37-abi3-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.7 abi3 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
f387715e934e63965f0b6f1972803cef9c31bef32d940a0a6681c1b23070dbc6
|
|
BLAKE2b-256 checksum How to use checksums |
f07ec8aacc08e0ab4fc5b83d6276a4f9d6383365b6589172f71926e2d35a744e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.16
|
