Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

curl_cffi

PyPI - Downloads PyPI - Python Version PyPI version Generic badge Generic badge

Documentation

Python binding for curl-impersonate fork via cffi.

Unlike other pure python http clients like httpx or requests, curl_cffi can impersonate browsers' TLS/JA3 and HTTP/2 fingerprints. If you are blocked by some website for no obvious reason, you can give curl_cffi a try.

Only Python 3.8 and above are supported. Python 3.7 has reached its end of life.


ProxyCurl

Scrape public LinkedIn profile data at scale with Proxycurl APIs. Built for developers, by developers.

  • GDPR, CCPA, SOC2 compliant
  • High rate limit (300 requests/min), Fast (APIs respond in ~2s), High accuracy
  • Fresh data - 88% of data is scraped real-time, other 12% is <29 days
  • Tons of data points returned per profile

Bypass Cloudflare with API

Yes Captcha!

Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g. cf_clearance). Click here to register: https://yescaptcha.com/i/stfnIO


Scrape Ninja

ScrapeNinja is a web scraping API with two engines: fast, with high performance and TLS fingerprint; and slower with a real browser under the hood.

ScrapeNinja handles headless browsers, proxies, timeouts, retries, and helps with data extraction, so you can just get the data in JSON. Rotating proxies are available out of the box on all subscription plans.


Features

  • Supports JA3/TLS and http2 fingerprints impersonation, including recent browsers and custome fingerprints.
  • Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
  • Mimics requests API, no need to learn another one.
  • Pre-compiled, so you don't have to compile on your machine.
  • Supports asyncio with proxy rotation on each request.
  • Supports http 2.0, which requests does not.
  • Supports websocket.
requests aiohttp httpx pycurl curl_cffi
http2 ❌ ❌ ✅ ✅ ✅
sync ✅ ❌ ✅ ✅ ✅
async ❌ ✅ ✅ ❌ ✅
websocket ❌ ✅ ❌ ❌ ✅
fingerprints ❌ ❌ ❌ ❌ ✅
speed 🐇 🐇🐇 🐇 🐇🐇 🐇🐇

Install

pip install curl_cffi --upgrade

This should work on Linux, macOS and Windows out of the box. If it does not work on you platform, you may need to compile and install curl-impersonate first and set some environment variables like LD_LIBRARY_PATH.

To install beta releases:

pip install curl_cffi --upgrade --pre

To install unstable version from GitHub:

git clone https://github.com/lexiforest/curl_cffi/
cd curl_cffi
make preprocess
pip install .

Usage

curl_cffi comes with a low-level curl API and a high-level requests-like API.

requests-like

from curl_cffi import requests

# Notice the impersonate parameter
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome")

print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser

# To keep using the latest browser version as `curl_cffi` updates,
# simply set impersonate="chrome" without specifying a version.
# Other similar values are: "safari" and "safari_ios"
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome")

# To pin a specific version, use version numbers together.
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome124")

# To impersonate other than browsers, bring your own ja3/akamai strings
# See examples directory for details.
r = requests.get("https://tls.browserleaks.com/json", ja3=..., akamai=...)

# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome", proxies=proxies)

proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tools.scrapfly.io/api/fp/ja3", impersonate="chrome", proxies=proxies)

Sessions

s = requests.Session()

# httpbin is a http test website, this endpoint makes the server set cookies
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>

# retrieve cookies again to verify
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}

curl_cffi supports the same browser versions as supported by my fork of curl-impersonate:

However, only WebKit-based browsers are supported. Firefox support is tracked in #59.

Browser versions will be added only when their fingerprints change. If you see a version, e.g. chrome122, were skipped, you can simply impersonate it with your own headers and the previous version.

If you are trying to impersonate a target other than a browser, use ja3=... and akamai=... to specify your own customized fingerprints. See the docs on impersonatation for details.

  • chrome99
  • chrome100
  • chrome101
  • chrome104
  • chrome107
  • chrome110
  • chrome116 [1]
  • chrome119 [1]
  • chrome120 [1]
  • chrome123 [3]
  • chrome124 [3]
  • chrome131 [4]
  • chrome99_android
  • chrome131_android [4]
  • edge99
  • edge101
  • safari15_3 [2]
  • safari15_5 [2]
  • safari17_0 [1]
  • safari17_2_ios [1]
  • safari18_0 [4]
  • safari18_0_ios [4]

Notes:

  1. Added in version 0.6.0.
  2. Fixed in version 0.6.0, previous http2 fingerprints were not correct.
  3. Added in version 0.7.0.
  4. Added in version 0.8.0.

asyncio

from curl_cffi.requests import AsyncSession

async with AsyncSession() as s:
    r = await s.get("https://example.com")

More concurrency:

import asyncio
from curl_cffi.requests import AsyncSession

urls = [
    "https://google.com/",
    "https://facebook.com/",
    "https://twitter.com/",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get(url)
        tasks.append(task)
    results = await asyncio.gather(*tasks)

WebSockets

from curl_cffi.requests import WebSocket

def on_message(ws: WebSocket, message: str | bytes):
    print(message)

ws = WebSocket(on_message=on_message)
ws.run_forever("wss://api.gemini.com/v1/marketdata/BTCUSD")

For low-level APIs, Scrapy integration and other advanced topics, see the docs for more details.

asyncio WebSockets

import asyncio
from curl_cffi.requests import AsyncSession

async with AsyncSession() as s:
    ws = await s.ws_connect("wss://echo.websocket.org")
    await asyncio.gather(*[ws.send_str("Hello, World!") for _ in range(10)])
    async for message in ws:
        print(message)

Acknowledgement

  • Originally forked from multippt/python_curl_cffi, which is under the MIT license.
  • Headers/Cookies files are copied from httpx, which is under the BSD license.
  • Asyncio support is inspired by Tornado's curl http client.
  • The synchronous WebSocket API is inspired by websocket_client.
  • The asynchronous WebSocket API is inspired by aiohttp.

Sponsor

Buy Me A Coffee

Metadata

Release files for curl-cffi 0.8.1b8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for curl-cffi 0.8.1b8
File Size Uploaded Provenance Actions
curl_cffi-0.8.1b8.tar.gz 143.7 kB Without provenance Details for curl_cffi-0.8.1b8.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for curl-cffi 0.8.1b8
File Actions
curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_x86_64.whl CPython 3.8 abi3 Linux musl 1.1+ x86-64 Details for curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_x86_64.whl
curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_aarch64.whl CPython 3.8 abi3 Linux musl 1.1+ ARM64 Details for curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_aarch64.whl
curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.8 abi3 Linux glibc 2.17+ x86-64 Details for curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl CPython 3.8 abi3 Linux glibc 2.17+ x86-32 Details for curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.8 abi3 Linux glibc 2.17+ ARM64 Details for curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
curl_cffi-0.8.1b8-cp38-abi3-macosx_11_0_arm64.whl CPython 3.8 abi3 macOS 11.0+ ARM64 Details for curl_cffi-0.8.1b8-cp38-abi3-macosx_11_0_arm64.whl
curl_cffi-0.8.1b8-cp38-abi3-macosx_10_9_x86_64.whl CPython 3.8 abi3 macOS 10.9+ x86-64 Details for curl_cffi-0.8.1b8-cp38-abi3-macosx_10_9_x86_64.whl

Total release size: 39.7 MB

Release files / curl_cffi-0.8.1b8.tar.gz

Download URL curl_cffi-0.8.1b8.tar.gz
Size 143.7 kB
Tags Source
SHA-256 checksum
How to use checksums
3a47342534bbdaa7459c0e0c5d440ab22956418b376ae82fbb6d8752303fbaf5
BLAKE2b-256 checksum
How to use checksums
adf845b7f8c044d7ac442dc3d021b45f2c7d465fcede94c7178f9f1a513b43d9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_x86_64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_x86_64.whl
Size 6.8 MB
Tags CPython 3.8 Linux musl 1.1+ x86-64 abi3
SHA-256 checksum
How to use checksums
674c91b0e18f3fcc7a32afa9e90917966fbe146c73df47bf7cc3b20203040bf4
BLAKE2b-256 checksum
How to use checksums
de0423132ea227767eb07a04e808fbb44d6d80d50e900f6e5044d3dec95025af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_aarch64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-musllinux_1_1_aarch64.whl
Size 6.8 MB
Tags CPython 3.8 Linux musl 1.1+ ARM64 abi3
SHA-256 checksum
How to use checksums
bfa80bd98b5452a8deeee65c1a78c44f197683e2d84baa9162b0c1eaef82330f
BLAKE2b-256 checksum
How to use checksums
1338473d68a109c0c9bad5cab9cb61ae0d62e6aa00b9401f21e5c5afde128a44
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 6.3 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
b7369f11edb0b44a5a4b761f573db5e17c271ccc0fc6a0ecd0d04d108cd3b680
BLAKE2b-256 checksum
How to use checksums
84f1d0812fe08adf1dbc77b95709f490c4d536496b6a9f4cba72c091ab731279
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Size 5.7 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-32 abi3
SHA-256 checksum
How to use checksums
cebee550353c9bc732538e4095e4cd191e2d52974b604ffbb5b4075a1d5d1a40
BLAKE2b-256 checksum
How to use checksums
2e1f7f3c4f3db364ed4f0c02fc3df3e627fcfc5baf4fac680c17c9508a9bdb34
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 5.9 MB
Tags CPython 3.8 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
6338e30c1010c6314f4b2b645790d565b3e1467ad4ea1d4a351bdcc1b32568c2
BLAKE2b-256 checksum
How to use checksums
99901718fffb7e593e153239c2ddb6baf2505e152720f9a00e489fe357e8c08e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-macosx_11_0_arm64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-macosx_11_0_arm64.whl
Size 2.6 MB
Tags CPython 3.8 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
bb93182d244eff952ad414c7efd0b2758e6a2b99c137ad283d3f71cace090c18
BLAKE2b-256 checksum
How to use checksums
7a5bc9a64f5d7f36a8db9ac39c1f5c8137bb9574f81f6853a8f5ab9ba21fa06f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Release files / curl_cffi-0.8.1b8-cp38-abi3-macosx_10_9_x86_64.whl

Download URL curl_cffi-0.8.1b8-cp38-abi3-macosx_10_9_x86_64.whl
Size 5.3 MB
Tags CPython 3.8 abi3 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
d077ad7616f6698fd2be24c77f571b2930551b1c86da8342cd60a92205ea1816
BLAKE2b-256 checksum
How to use checksums
79faa05b998dd95b34ade84bf926e80cb8ee8fdced44d4935a2aef330eec28be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.8

Security for curl-cffi 0.8.1b8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page