HTTP client that can impersonate web browsers, mimicking their headers and `TLS/JA3/JA4/HTTP2` fingerprints
Project description
Pyreqwest_impersonate
The fastest python HTTP client that can impersonate web browsers.
Provides precompiled wheels: 🐧linux|musllinux
(amd64, aarch64), 🪟windows
(amd64); 🍏macos
(amd64, aarch64).
Table of Contents
Installation
pip install -U pyreqwest_impersonate
Features
- Impersonate: impersonate option mimics web browsers by replicating their headers and TLS/JA3/JA4/HTTP2 fingerprints.
- Automatic Character Encoding Detection: the encoding is taken from the
Content-Type
header or<meta ... charset=
within the html, or if not found,UTF-8
. If the encoding does not match the content, the library automatically detects and uses the correct encoding to decode the text. - Html2text: the
text_plain
|text_markdown
response attributes extract plain|markdown text from html. - All decoders:
gzip
,brotli
,zstd
are already included and do not require third party packages. - Small Size: the compiled library is about 6.9MB in size.
- High Performance: the library is designed for a large number of threads, uses all processors, and releases the GIL.
- Thread-safe: library can be safely used in multithreaded environments.
Benchmark
Usage
I. Client
HTTP client that can impersonate web browsers.
class Client:
"""Initializes an HTTP client that can impersonate web browsers.
Args:
auth (tuple, optional): A tuple containing the username and password for basic authentication. Default is None.
auth_bearer (str, optional): Bearer token for authentication. Default is None.
params (dict, optional): Default query parameters to include in all requests. Default is None.
headers (dict, optional): Default headers to send with requests. If `impersonate` is set, this will be ignored.
cookies (dict, optional): - An optional map of cookies to send with requests as the `Cookie` header.
timeout (float, optional): HTTP request timeout in seconds. Default is 30.
cookie_store (bool, optional): Enable a persistent cookie store. Received cookies will be preserved and included
in additional requests. Default is True.
referer (bool, optional): Enable or disable automatic setting of the `Referer` header. Default is True.
proxy (str, optional): Proxy URL for HTTP requests. Example: "socks5://127.0.0.1:9150". Default is None.
impersonate (str, optional): Entity to impersonate. Example: "chrome_124". Default is None.
Chrome: "chrome_99","chrome_100","chrome_101","chrome_104","chrome_105","chrome_106","chrome_107",
"chrome_108","chrome_109","chrome_114","chrome_116","chrome_117","chrome_118","chrome_119",
"chrome_120","chrome_123","chrome_124","chrome_126"
Safari: "safari_ios_16.5","safari_ios_17.2","safari_ios_17.4.1","safari_15.3","safari_15.5","safari_15.6.1",
"safari_16","safari_16.5","safari_17.2.1","safari_17.4.1","safari_17.5"
OkHttp: "okhttp_3.9","okhttp_3.11","okhttp_3.13","okhttp_3.14","okhttp_4.9","okhttp_4.10","okhttp_5"
Edge: "edge_99","edge_101","edge_122"
follow_redirects (bool, optional): Whether to follow redirects. Default is True.
max_redirects (int, optional): Maximum redirects to follow. Default 20. Applies if `follow_redirects` is True.
verify (bool, optional): Verify SSL certificates. Default is True.
http1 (bool, optional): Use only HTTP/1.1. Default is None.
http2 (bool, optional): Use only HTTP/2. Default is None.
"""
Client methods
The Client
class provides a set of methods for making HTTP requests: get
, head
, options
, delete
, post
, put
, patch
, each of which internally utilizes the request()
method for execution. The parameters for these methods closely resemble those in httpx
.
def get(
url: str,
params: Optional[Dict[str, str]] = None,
headers: Optional[Dict[str, str]] = None,
cookies: Optional[Dict[str, str]] = None,
auth: Optional[Tuple[str, Optional[str]]] = None,
auth_bearer: Optional[str] = None,
timeout: Optional[float] = 30,
):
"""Performs a GET request to the specified URL.
Args:
url (str): The URL to which the request will be made.
params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password
for basic authentication. Default is None.
auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
timeout (Optional[float]): The timeout for the request in seconds. Default is 30.
"""
def post(
url: str,
params: Optional[Dict[str, str]] = None,
headers: Optional[Dict[str, str]] = None,
cookies: Optional[Dict[str, str]] = None,
content: Optional[bytes] = None,
data: Optional[Dict[str, str]] = None,
json: Any = None,
files: Optional[Dict[str, bytes]] = None,
auth: Optional[Tuple[str, Optional[str]]] = None,
auth_bearer: Optional[str] = None,
timeout: Optional[float] = 30,
):
"""Performs a POST request to the specified URL.
Args:
url (str): The URL to which the request will be made.
params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
content (Optional[bytes]): The content to send in the request body as bytes. Default is None.
data (Optional[Dict[str, str]]): The form data to send in the request body. Default is None.
json (Any): A JSON serializable object to send in the request body. Default is None.
files (Optional[Dict[str, bytes]]): A map of file fields to file contents to be sent as multipart/form-data. Default is None.
auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password
for basic authentication. Default is None.
auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
timeout (Optional[float]): The timeout for the request in seconds. Default is 30.
"""
Response object
resp.content
resp.cookies
resp.encoding
resp.headers
resp.json()
resp.status_code
resp.text
resp.text_markdown # html is converted to markdown text
resp.text_plain # html is converted to plain text
resp.url
Examples
import pyreqwest_impersonate as pri
client = pri.Client(impersonate="chrome_126")
# GET request
resp = client.get("https://tls.peet.ws/api/all")
print(resp.json())
# GET request with passing params and setting timeout
params = {"param1": "value1", "param2": "value2"}
resp = client.post(url="https://httpbin.org/anything", params=params, timeout=10)
print(r.text)
# POST Binary Request Data
content = b"some_data"
resp = client.post(url="https://httpbin.org/anything", content=content)
print(r.text)
# POST Form Encoded Data
data = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", data=data)
print(r.text)
# POST JSON Encoded Data
json = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", json=json)
print(r.text)
# POST Multipart-Encoded Files
files = {'file1': open('file1.txt', 'rb').read(), 'file2': open('file2.txt', 'rb').read()}
r = client.post("https://httpbin.org/post", files=files)
print(r.text)
# Authentication using user/password
auth = ("user", "password")
resp = client.post(url="https://httpbin.org/anything", auth=auth)
print(r.text)
# Authentication using auth bearer
auth_bearer = "bearerXXXXXXXXXXXXXXXXXXXX"
resp = client.post(url="https://httpbin.org/anything", auth_bearer=auth_bearer)
print(r.text)
# Using proxy
resp = pri.Client(proxy="http://127.0.0.1:8080").get("https://tls.peet.ws/api/all")
print(resp.json())
# You can also use convenience functions that use a default Client instance under the hood:
# pri.get() | pri.head() | pri.options() | pri.delete() | pri.post() | pri.patch() | pri.put()
# These functions can accept the `impersonate` parameter:
resp = pri.get("https://httpbin.org/anything", impersonate="chrome_126")
print(r.text)
II. AsyncClient
TODO
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file pyreqwest_impersonate-0.5.3.tar.gz
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3.tar.gz
- Upload date:
- Size: 79.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f21c10609958ff5be18df0c329eed42d2b3ba8a339b65dc5f96ab74537231692 |
|
MD5 | 44fdf54ceeacfdee7b9ab727d555971d |
|
BLAKE2b-256 | d5fa1907e6ab36f51dcfdcd89905941b994b78a3c6609a11884b653ebe7a9ea3 |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl
- Upload date:
- Size: 3.0 MB
- Tags: CPython 3.8+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6026e4751b5912aec1e45238c07daf1e2c9126b3b32b33396b72885021e8990c |
|
MD5 | 91eeb3d04e319cef189989330bdc8e0e |
|
BLAKE2b-256 | 0419413f07ecfa18382ac48d4a0f01a17aeb0a9d124b78d9e3ebaca534e413c8 |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl
- Upload date:
- Size: 3.0 MB
- Tags: CPython 3.8+, musllinux: musl 1.2+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7397f6dad3d5ae158e0b272cb3eafe8382e71775d829b286ae9c21cb5a879ff |
|
MD5 | 058c462fcf30a14653116c64c6c9bf25 |
|
BLAKE2b-256 | 244dedddabb322a69e042f2b85d17163c847080d6b8fee4789238108197fdfef |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl
- Upload date:
- Size: 2.8 MB
- Tags: CPython 3.8+, musllinux: musl 1.2+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6bf986d4a165f6976b3e862111e2a46091883cb55e9e6325150f5aea2644229 |
|
MD5 | d10779efbe0c890a70104d0c34ead021 |
|
BLAKE2b-256 | 1e7c9e6a5fe7219595387261fa455276f95716c3ed81bdbbf1998e82366a3d99 |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 3.0 MB
- Tags: CPython 3.8+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9d2e981a525fb72c1521f454e5581d2c7a3b1fcf1c97c0acfcb7a923d8cf3e |
|
MD5 | 53202c47062490c4238703b5bb7d3fdb |
|
BLAKE2b-256 | 1fec95514b593277c77e371292001cc1837790362749d5839f4e4e5f1e0d7a20 |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 2.8 MB
- Tags: CPython 3.8+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 436055fa3eeb3e01e2e8efd42a9f6c4ab62fd643eddc7c66d0e671b71605f273 |
|
MD5 | 90b9b8f7c873c8f9b4c23a2de4dca23f |
|
BLAKE2b-256 | 2745446787685472687cfc2c2ff7ba0c520978b2ee96808ec7c7057e142c760e |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 2.7 MB
- Tags: CPython 3.8+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77533133ae73020e59bc56d776eea3fe3af4ac41d763a89f39c495436da0f4cf |
|
MD5 | 8e42755118d5492f21024ea83faf8858 |
|
BLAKE2b-256 | ebe190085ca84f5894469e26238a46f9a22b631897278beb45bbabbaa6450914 |
File details
Details for the file pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl
.
File metadata
- Download URL: pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 2.9 MB
- Tags: CPython 3.8+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.7.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f15922496f728769fb9e1b116d5d9d7ba5525d0f2f7a76a41a1daef8b2e0c6c3 |
|
MD5 | 84723948f0d920e96108fcbf8425f678 |
|
BLAKE2b-256 | 34b88ab81ac9394ffb1aeea1fb0894c74e3b16d69a3c0005e0958f1c6df56971 |