HTTP client that can impersonate web browsers, mimicking their headers and `TLS/JA3/JA4/HTTP2` fingerprints
Project description
Pyreqwest_impersonate
The fastest python HTTP client that can impersonate web browsers.
Provides precompiled wheels: 🐧linux|musllinux
(amd64, aarch64), 🪟windows
(amd64); 🍏macos
(amd64, aarch64).
Table of Contents
Installation
pip install -U pyreqwest_impersonate
Features
- Impersonate: impersonate option mimics web browsers by replicating their headers and TLS/JA3/JA4/HTTP2 fingerprints.
- Automatic Character Encoding Detection: the encoding is taken from the
Content-Type
header or<meta ... charset=
within the html, or if not found,UTF-8
. If the encoding does not match the content, the library automatically detects and uses the correct encoding to decode the text. - Html2text: the
text_plain
|text_markdown
response attributes extract plain|markdown text from html. - All decoders:
gzip
,brotli
,zstd
are already included and do not require third party packages. - Small Size: the compiled library is about 6.9MB in size.
- High Performance: the library is designed for a large number of threads, uses all processors, and releases the GIL.
- Thread-safe: library can be safely used in multithreaded environments.
Benchmark
Usage
I. Client
HTTP client that can impersonate web browsers.
class Client:
"""Initializes an HTTP client that can impersonate web browsers.
Args:
auth (tuple, optional): A tuple containing the username and password for basic authentication. Default is None.
auth_bearer (str, optional): Bearer token for authentication. Default is None.
params (dict, optional): Default query parameters to include in all requests. Default is None.
headers (dict, optional): Default headers to send with requests. If `impersonate` is set, this will be ignored.
cookies (dict, optional): - An optional map of cookies to send with requests as the `Cookie` header.
timeout (float, optional): HTTP request timeout in seconds. Default is 30.
cookie_store (bool, optional): Enable a persistent cookie store. Received cookies will be preserved and included
in additional requests. Default is True.
referer (bool, optional): Enable or disable automatic setting of the `Referer` header. Default is True.
proxy (str, optional): Proxy URL for HTTP requests. Example: "socks5://127.0.0.1:9150". Default is None.
impersonate (str, optional): Entity to impersonate. Example: "chrome_124". Default is None.
Chrome: "chrome_99","chrome_100","chrome_101","chrome_104","chrome_105","chrome_106","chrome_107",
"chrome_108","chrome_109","chrome_114","chrome_116","chrome_117","chrome_118","chrome_119",
"chrome_120","chrome_123","chrome_124","chrome_126"
Safari: "safari_ios_16.5","safari_ios_17.2","safari_ios_17.4.1","safari_15.3","safari_15.5","safari_15.6.1",
"safari_16","safari_16.5","safari_17.2.1","safari_17.4.1","safari_17.5"
OkHttp: "okhttp_3.9","okhttp_3.11","okhttp_3.13","okhttp_3.14","okhttp_4.9","okhttp_4.10","okhttp_5"
Edge: "edge_99","edge_101","edge_122"
follow_redirects (bool, optional): Whether to follow redirects. Default is True.
max_redirects (int, optional): Maximum redirects to follow. Default 20. Applies if `follow_redirects` is True.
verify (bool, optional): Verify SSL certificates. Default is True.
http1 (bool, optional): Use only HTTP/1.1. Default is None.
http2 (bool, optional): Use only HTTP/2. Default is None.
"""
Client methods
The Client
class provides a set of methods for making HTTP requests: get
, head
, options
, delete
, post
, put
, patch
, each of which internally utilizes the request()
method for execution. The parameters for these methods closely resemble those in httpx
.
def get(
url: str,
params: Optional[Dict[str, str]] = None,
headers: Optional[Dict[str, str]] = None,
cookies: Optional[Dict[str, str]] = None,
auth: Optional[Tuple[str, Optional[str]]] = None,
auth_bearer: Optional[str] = None,
timeout: Optional[float] = 30,
):
"""Performs a GET request to the specified URL.
Args:
url (str): The URL to which the request will be made.
params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password
for basic authentication. Default is None.
auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
timeout (Optional[float]): The timeout for the request in seconds. Default is 30.
"""
def post(
url: str,
params: Optional[Dict[str, str]] = None,
headers: Optional[Dict[str, str]] = None,
cookies: Optional[Dict[str, str]] = None,
content: Optional[bytes] = None,
data: Optional[Dict[str, str]] = None,
json: Any = None,
files: Optional[Dict[str, bytes]] = None,
auth: Optional[Tuple[str, Optional[str]]] = None,
auth_bearer: Optional[str] = None,
timeout: Optional[float] = 30,
):
"""Performs a POST request to the specified URL.
Args:
url (str): The URL to which the request will be made.
params (Optional[Dict[str, str]]): A map of query parameters to append to the URL. Default is None.
headers (Optional[Dict[str, str]]): A map of HTTP headers to send with the request. Default is None.
cookies (Optional[Dict[str, str]]): - An optional map of cookies to send with requests as the `Cookie` header.
content (Optional[bytes]): The content to send in the request body as bytes. Default is None.
data (Optional[Dict[str, str]]): The form data to send in the request body. Default is None.
json (Any): A JSON serializable object to send in the request body. Default is None.
files (Optional[Dict[str, bytes]]): A map of file fields to file contents to be sent as multipart/form-data. Default is None.
auth (Optional[Tuple[str, Optional[str]]]): A tuple containing the username and an optional password
for basic authentication. Default is None.
auth_bearer (Optional[str]): A string representing the bearer token for bearer token authentication. Default is None.
timeout (Optional[float]): The timeout for the request in seconds. Default is 30.
"""
Response object
resp.content
resp.cookies
resp.encoding
resp.headers
resp.json()
resp.status_code
resp.text
resp.text_markdown # html is converted to markdown text
resp.text_plain # html is converted to plain text
resp.url
Examples
import pyreqwest_impersonate as pri
client = pri.Client(impersonate="chrome_126")
# GET request
resp = client.get("https://tls.peet.ws/api/all")
print(resp.json())
# GET request with passing params and setting timeout
params = {"param1": "value1", "param2": "value2"}
resp = client.post(url="https://httpbin.org/anything", params=params, timeout=10)
print(r.text)
# POST Binary Request Data
content = b"some_data"
resp = client.post(url="https://httpbin.org/anything", content=content)
print(r.text)
# POST Form Encoded Data
data = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", data=data)
print(r.text)
# POST JSON Encoded Data
json = {"key1": "value1", "key2": "value2"}
resp = client.post(url="https://httpbin.org/anything", json=json)
print(r.text)
# POST Multipart-Encoded Files
files = {'file1': open('file1.txt', 'rb').read(), 'file2': open('file2.txt', 'rb').read()}
r = client.post("https://httpbin.org/post", files=files)
print(r.text)
# Authentication using user/password
auth = ("user", "password")
resp = client.post(url="https://httpbin.org/anything", auth=auth)
print(r.text)
# Authentication using auth bearer
auth_bearer = "bearerXXXXXXXXXXXXXXXXXXXX"
resp = client.post(url="https://httpbin.org/anything", auth_bearer=auth_bearer)
print(r.text)
# Using proxy
resp = pri.Client(proxy="http://127.0.0.1:8080").get("https://tls.peet.ws/api/all")
print(resp.json())
# You can also use convenience functions that use a default Client instance under the hood:
# pri.get() | pri.head() | pri.options() | pri.delete() | pri.post() | pri.patch() | pri.put()
# These functions can accept the `impersonate` parameter:
resp = pri.get("https://httpbin.org/anything", impersonate="chrome_126")
print(r.text)
II. AsyncClient
TODO
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Close
Hashes for pyreqwest_impersonate-0.5.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f21c10609958ff5be18df0c329eed42d2b3ba8a339b65dc5f96ab74537231692 |
|
MD5 | 44fdf54ceeacfdee7b9ab727d555971d |
|
BLAKE2b-256 | d5fa1907e6ab36f51dcfdcd89905941b994b78a3c6609a11884b653ebe7a9ea3 |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6026e4751b5912aec1e45238c07daf1e2c9126b3b32b33396b72885021e8990c |
|
MD5 | 91eeb3d04e319cef189989330bdc8e0e |
|
BLAKE2b-256 | 0419413f07ecfa18382ac48d4a0f01a17aeb0a9d124b78d9e3ebaca534e413c8 |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7397f6dad3d5ae158e0b272cb3eafe8382e71775d829b286ae9c21cb5a879ff |
|
MD5 | 058c462fcf30a14653116c64c6c9bf25 |
|
BLAKE2b-256 | 244dedddabb322a69e042f2b85d17163c847080d6b8fee4789238108197fdfef |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6bf986d4a165f6976b3e862111e2a46091883cb55e9e6325150f5aea2644229 |
|
MD5 | d10779efbe0c890a70104d0c34ead021 |
|
BLAKE2b-256 | 1e7c9e6a5fe7219595387261fa455276f95716c3ed81bdbbf1998e82366a3d99 |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9d2e981a525fb72c1521f454e5581d2c7a3b1fcf1c97c0acfcb7a923d8cf3e |
|
MD5 | 53202c47062490c4238703b5bb7d3fdb |
|
BLAKE2b-256 | 1fec95514b593277c77e371292001cc1837790362749d5839f4e4e5f1e0d7a20 |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 436055fa3eeb3e01e2e8efd42a9f6c4ab62fd643eddc7c66d0e671b71605f273 |
|
MD5 | 90b9b8f7c873c8f9b4c23a2de4dca23f |
|
BLAKE2b-256 | 2745446787685472687cfc2c2ff7ba0c520978b2ee96808ec7c7057e142c760e |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77533133ae73020e59bc56d776eea3fe3af4ac41d763a89f39c495436da0f4cf |
|
MD5 | 8e42755118d5492f21024ea83faf8858 |
|
BLAKE2b-256 | ebe190085ca84f5894469e26238a46f9a22b631897278beb45bbabbaa6450914 |
Close
Hashes for pyreqwest_impersonate-0.5.3-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f15922496f728769fb9e1b116d5d9d7ba5525d0f2f7a76a41a1daef8b2e0c6c3 |
|
MD5 | 84723948f0d920e96108fcbf8425f678 |
|
BLAKE2b-256 | 34b88ab81ac9394ffb1aeea1fb0894c74e3b16d69a3c0005e0958f1c6df56971 |