Skip to main content

HTTP client that can impersonate web browsers, mimicking their headers and `TLS/JA3/JA4/HTTP2` fingerprints

Project description

Python >= 3.8 Downloads CI

Pyreqwest_impersonate

The fastest python HTTP client that can impersonate web browsers.
Provides precompiled wheels: 🐧linux|musllinux (amd64, aarch64), 🪟windows (amd64); 🍏macos (amd64, aarch64).

Table of Contents

Installation

pip install -U pyreqwest_impersonate

Features

  • Impersonate: impersonate option mimics web browsers by replicating their headers and TLS/JA3/JA4/HTTP2 fingerprints.
  • Automatic Character Encoding Detection: the encoding is taken from the Content-Type header or <meta ... charset= within the html, or if not found, UTF-8. If the encoding does not match the content, the library automatically detects and uses the correct encoding to decode the text.
  • Html2text: the text_plain|text_markdown response attributes extract plain|markdown text from html.
  • All decoders: gzip, brotli, zstd are already included and do not require third party packages.
  • Small Size: the compiled library is about 6.9MB in size.
  • High Performance: the library is designed for a large number of threads, uses all processors, and releases the GIL.
  • Thread-safe: library can be safely used in multithreaded environments.

Benchmark

Usage

I. Client

HTTP client that can impersonate web browsers.

class Client:
    """Initializes an HTTP client that can impersonate web browsers.
    
    Args:
        auth (tuple, optional): A tuple containing the username and password for basic authentication. Default is None.
        auth_bearer (str, optional): Bearer token for authentication. Default is None.
        params (dict, optional): Default query parameters to include in all requests. Default is None.
        headers (dict, optional): Default headers to send with requests. If `impersonate` is set, this will be ignored.
        cookies (dict, optional): - An optional map of cookies to send with requests as the `Cookie` header.
        timeout (float, optional): HTTP request timeout in seconds. Default is 30.
        cookie_store (bool, optional): Enable a persistent cookie store. Received cookies will be preserved and included 
            in additional requests. Default is True.
        referer (bool, optional): Enable or disable automatic setting of the `Referer` header. Default is True.
        proxy (str, optional): Proxy URL for HTTP requests. Example: "socks5://127.0.0.1:9150". Default is None.
        impersonate (str, optional): Entity to impersonate. Example: "chrome_124". Default is None.
            Chrome: "chrome_99","chrome_100","chrome_101","chrome_104","chrome_105","chrome_106","chrome_107", 
                "chrome_108","chrome_109","chrome_114","chrome_116","chrome_117","chrome_118","chrome_119", 
                "chrome_120","chrome_123","chrome_124","chrome_126"
            Safari: "safari_ios_16.5","safari_ios_17.2","safari_ios_17.4.1","safari_15.3","safari_15.5","safari_15.6.1",
                "safari_16","safari_16.5","safari_17.2.1","safari_17.4.1","safari_17.5"
            OkHttp: "okhttp_3.9","okhttp_3.11","okhttp_3.13","okhttp_3.14","okhttp_4.9","okhttp_4.10","okhttp_5"
            Edge: "edge_99","edge_101","edge_122"
        follow_redirects (bool, optional): Whether to follow redirects. Default is True.
        max_redirects (int, optional): Maximum redirects to follow. Default 20. Applies if `follow_redirects` is True.
        verify (bool, optional): Verify SSL certificates. Default is True.
        http1 (bool, optional): Use only HTTP/1.1. Default is None.
        http2 (bool, optional): Use only HTTP/2. Default is None.
         
    """

Client methods

The Client class provides a set of methods for making HTTP requests: get, head, options, delete, post, put, patch, each of which internally utilizes the request() method for execution. The parameters for these methods closely resemble those in httpx.

def get(
    url: str, 
    params: Optional[Dict[str, str]] = None, 
    headers: Optional[Dict[str, str]] = None, 
    cookies: Optional[Dict[str, str]] = None, 
    auth: Optional[Tuple[str, Optional[str]]] = None, 
    auth_bearer: Optional[str] = None, 
    timeout: Optional[float] = 30,
):
    """Performs a GET request to the specified URL.

    Args:
        url (str): The URL to which the request will be made.
        params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
        headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
        cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
        auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password 
            for basic authentication. Default is None.
        auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
        timeout (Optional[float]): The timeout for the request in seconds. Default is 30.

    """
def post(
    url: str, 
    params: Optional[Dict[str, str]] = None, 
    headers: Optional[Dict[str, str]] = None, 
    cookies: Optional[Dict[str, str]] = None, 
    content: Optional[bytes] = None, 
    data: Optional[Dict[str, str]] = None, 
    json: Any = None, 
    files: Optional[Dict[str, bytes]] = None, 
    auth: Optional[Tuple[str, Optional[str]]] = None, 
    auth_bearer: Optional[str] = None, 
    timeout: Optional[float] = 30,
):
    """Performs a POST request to the specified URL.

    Args:
        url (str): The URL to which the request will be made.
        params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
        headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
        cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
        content (Optional[bytes]): The content to send in the request body as bytes. Default is None.
        data (Optional[Dict[str, str]]): The form data to send in the request body. Default is None.
        json (Any): A JSON serializable object to send in the request body. Default is None.
        files (Optional[Dict[str, bytes]]): A map of file fields to file contents to be sent as multipart/form-data. Default is None.
        auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password 
            for basic authentication. Default is None.
        auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
        timeout (Optional[float]): The timeout for the request in seconds. Default is 30.

    """

Response object

resp.content
resp.cookies
resp.encoding
resp.headers
resp.json()
resp.status_code
resp.text
resp.text_markdown  # html is converted to markdown text
resp.text_plain  # html is converted to plain text
resp.url

Examples

import pyreqwest_impersonate as pri

client = pri.Client(impersonate="chrome_126")

# GET request
resp = client.get("https://tls.peet.ws/api/all")
print(resp.json())

# GET request with passing params and setting timeout
params = {"param1": "value1", "param2": "value2"}
resp = client.post(url="https://httpbin.org/anything", params=params, timeout=10)
print(r.text)

# POST Binary Request Data
content = b"some_data"
resp = client.post(url="https://httpbin.org/anything", content=content)
print(r.text)

# POST Form Encoded Data
data = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", data=data)
print(r.text)

# POST JSON Encoded Data
json = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", json=json)
print(r.text)

# POST Multipart-Encoded Files
files = {'file1': open('file1.txt', 'rb').read(), 'file2': open('file2.txt', 'rb').read()}
r = client.post("https://httpbin.org/post", files=files)
print(r.text)

# Authentication using user/password
auth = ("user", "password")
resp = client.post(url="https://httpbin.org/anything", auth=auth)
print(r.text)

# Authentication using auth bearer
auth_bearer = "bearerXXXXXXXXXXXXXXXXXXXX"
resp = client.post(url="https://httpbin.org/anything", auth_bearer=auth_bearer)
print(r.text)

# Using proxy
resp = pri.Client(proxy="http://127.0.0.1:8080").get("https://tls.peet.ws/api/all")
print(resp.json())

# You can also use convenience functions that use a default Client instance under the hood:
# pri.get() | pri.head() | pri.options() | pri.delete() | pri.post() | pri.patch() | pri.put()
# These functions can accept the `impersonate` parameter:
resp = pri.get("https://httpbin.org/anything", impersonate="chrome_126")
print(r.text)

II. AsyncClient

TODO

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyreqwest_impersonate-0.5.3.tar.gz (79.3 kB view details)

Uploaded Source

Built Distributions

pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl (3.0 MB view details)

Uploaded CPython 3.8+ Windows x86-64

pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl (3.0 MB view details)

Uploaded CPython 3.8+ musllinux: musl 1.2+ x86-64

pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.8+ musllinux: musl 1.2+ ARM64

pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB view details)

Uploaded CPython 3.8+ manylinux: glibc 2.17+ x86-64

pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.8+ manylinux: glibc 2.17+ ARM64

pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl (2.7 MB view details)

Uploaded CPython 3.8+ macOS 11.0+ ARM64

pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.8+ macOS 10.12+ x86-64

File details

Details for the file pyreqwest_impersonate-0.5.3.tar.gz.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3.tar.gz
Algorithm Hash digest
SHA256 f21c10609958ff5be18df0c329eed42d2b3ba8a339b65dc5f96ab74537231692
MD5 44fdf54ceeacfdee7b9ab727d555971d
BLAKE2b-256 d5fa1907e6ab36f51dcfdcd89905941b994b78a3c6609a11884b653ebe7a9ea3

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 6026e4751b5912aec1e45238c07daf1e2c9126b3b32b33396b72885021e8990c
MD5 91eeb3d04e319cef189989330bdc8e0e
BLAKE2b-256 0419413f07ecfa18382ac48d4a0f01a17aeb0a9d124b78d9e3ebaca534e413c8

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 b7397f6dad3d5ae158e0b272cb3eafe8382e71775d829b286ae9c21cb5a879ff
MD5 058c462fcf30a14653116c64c6c9bf25
BLAKE2b-256 244dedddabb322a69e042f2b85d17163c847080d6b8fee4789238108197fdfef

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 a6bf986d4a165f6976b3e862111e2a46091883cb55e9e6325150f5aea2644229
MD5 d10779efbe0c890a70104d0c34ead021
BLAKE2b-256 1e7c9e6a5fe7219595387261fa455276f95716c3ed81bdbbf1998e82366a3d99

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7e9d2e981a525fb72c1521f454e5581d2c7a3b1fcf1c97c0acfcb7a923d8cf3e
MD5 53202c47062490c4238703b5bb7d3fdb
BLAKE2b-256 1fec95514b593277c77e371292001cc1837790362749d5839f4e4e5f1e0d7a20

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 436055fa3eeb3e01e2e8efd42a9f6c4ab62fd643eddc7c66d0e671b71605f273
MD5 90b9b8f7c873c8f9b4c23a2de4dca23f
BLAKE2b-256 2745446787685472687cfc2c2ff7ba0c520978b2ee96808ec7c7057e142c760e

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 77533133ae73020e59bc56d776eea3fe3af4ac41d763a89f39c495436da0f4cf
MD5 8e42755118d5492f21024ea83faf8858
BLAKE2b-256 ebe190085ca84f5894469e26238a46f9a22b631897278beb45bbabbaa6450914

See more details on using hashes here.

File details

Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 f15922496f728769fb9e1b116d5d9d7ba5525d0f2f7a76a41a1daef8b2e0c6c3
MD5 84723948f0d920e96108fcbf8425f678
BLAKE2b-256 34b88ab81ac9394ffb1aeea1fb0894c74e3b16d69a3c0005e0958f1c6df56971

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page