Skip to main content

libcurl ffi bindings for Python, with impersonation support

Project description

curl_cffi

Python binding for curl-impersonate via cffi.

Documentation | 中文 README

Unlike other pure python http clients like httpx or requests, curl_cffi can impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some website for no obvious reason, you can give this package a try.

Features

  • Supports JA3/TLS and http2 fingerprints impersonation.
  • Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
  • Mimics requests API, no need to learn another one.
  • Pre-compiled, so you don't have to compile on your machine.
  • Supports asyncio with proxy rotation on each request.
  • Supports http 2.0, which requests does not.
library requests aiohttp httpx pycurl curl_cffi
http2
sync
async
fingerprints
speed 🐇 🐇🐇 🐇 🐇🐇 🐇🐇

Install

pip install curl_cffi --upgrade

This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64). If it does not work on you platform, you may need to compile and install curl-impersonate first and set some environment variables like LD_LIBRARY_PATH.

To install beta releases:

pip install curl_cffi --pre

Usage

requests-like

from curl_cffi import requests

# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")

print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser

# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)

proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)

Sessions

# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}

Supported impersonate versions, as supported by curl-impersonate:

  • chrome99
  • chrome100
  • chrome101
  • chrome104
  • chrome107
  • chrome110
  • chrome99_android
  • edge99
  • edge101
  • safari15_3
  • safari15_5

asyncio

from curl_cffi.requests import AsyncSession

async with AsyncSession() as s:
    r = await s.get("https://example.com")

More concurrency:

import asyncio
from curl_cffi.requests import AsyncSession

urls = [
    "https://googel.com/",
    "https://facebook.com/",
    "https://twitter.com/",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get("https://example.com")
        tasks.append(task)
    results = await asyncio.gather(*tasks)

curl-like

Alternatively, you can use the low-level curl-like API:

from curl_cffi import Curl, CurlOpt
from io import BytesIO

buffer = BytesIO()
c = Curl()
c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
c.setopt(CurlOpt.WRITEDATA, buffer)

c.impersonate("chrome110")

c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())

See the docs for more details.

If you are using scrapy, check out this middleware: tieyongjie/scrapy-fingerprint

Acknowledgement

  • Originally forked from multippt/python_curl_cffi, which is under the MIT license.
  • Headers/Cookies files are copied from httpx, which is under the BSD license.
  • Asyncio support is inspired by Tornado's curl http client.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

curl_cffi-0.5.10.tar.gz (35.2 kB view details)

Uploaded Source

Built Distributions

curl_cffi-0.5.10-cp37-abi3-win_amd64.whl (2.6 MB view details)

Uploaded CPython 3.7+Windows x86-64

curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.2 MB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ x86-64

curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.9 MB view details)

Uploaded CPython 3.7+manylinux: glibc 2.17+ ARM64

curl_cffi-0.5.10-cp37-abi3-macosx_11_0_arm64.whl (2.2 MB view details)

Uploaded CPython 3.7+macOS 11.0+ ARM64

curl_cffi-0.5.10-cp37-abi3-macosx_10_9_x86_64.whl (4.9 MB view details)

Uploaded CPython 3.7+macOS 10.9+ x86-64

File details

Details for the file curl_cffi-0.5.10.tar.gz.

File metadata

  • Download URL: curl_cffi-0.5.10.tar.gz
  • Upload date:
  • Size: 35.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for curl_cffi-0.5.10.tar.gz
Algorithm Hash digest
SHA256 55bac4b73e2d80ceeaabea33270fc8ca6ace594128a46710242f2e688b4f8bfc
MD5 e9324563a70356c82a766324e927a003
BLAKE2b-256 505743d0e4475c336282deb7acf5285c35fbdce6a041d1583dbe30e0d761117d

See more details on using hashes here.

File details

Details for the file curl_cffi-0.5.10-cp37-abi3-win_amd64.whl.

File metadata

  • Download URL: curl_cffi-0.5.10-cp37-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.7+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for curl_cffi-0.5.10-cp37-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 f9a1874b860c4e8db49bdfd9b9d4dc39999a1397d271ec78624c35c838e9e92a
MD5 341868ec10c7892bf44ce6275b1374cd
BLAKE2b-256 4e700d4d667287d54cb4daf8ce67068565c8e09e4f839a835119cd27f229a446

See more details on using hashes here.

File details

Details for the file curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1b2bc8822d23415f6533c8b750475e9bbc76764025fe1dcb5866dc033607fd7b
MD5 3f3678fadcdad60d9324b728047275ad
BLAKE2b-256 93040d57a51690451437f72563031036ea4f4b529bc0cfd8e8d93487075cbe80

See more details on using hashes here.

File details

Details for the file curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.5.10-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b537595b9610a4dd0927c09823925b4e32b1ce0fd04385bfc5bb72ab830720e6
MD5 0982e65b8d4553488596fc9b79dc88b6
BLAKE2b-256 162edf2c475bae057af3fff4d288a388062032e7005ad312e8f55490faef82e2

See more details on using hashes here.

File details

Details for the file curl_cffi-0.5.10-cp37-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.5.10-cp37-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9937b8e13b1a6963c63e155b6621ec74649965105efedb919bc226fe731861cc
MD5 a2c356014da1d4b9ec0053f6729f1efe
BLAKE2b-256 23253c9fdb7e08e4722eb9e7ebcc59c74d7eaf32ca4a878a3fdf581030940431

See more details on using hashes here.

File details

Details for the file curl_cffi-0.5.10-cp37-abi3-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for curl_cffi-0.5.10-cp37-abi3-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 892603dab5e56fb72bfff7ae969136138971f63f63defe98232e1ec55cb0f1c6
MD5 cb3877f3efbf7c389781131af43a356d
BLAKE2b-256 301872d6d4f5ec0db3ea68dde9ad775f9985ce22e699a47ab5b4cfcf8d74f3f5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page