human-requests 0.1.5.1

Playwright wrapper обеспечивающий необходимый функционал для человеко-подобного парсинга сайтов

🧰 Human Requests

Asynchronous library for browser‑like HTTP scenarios with controlled offline rendering and two‑way state transfer.

⭐ Star us on GitHub | 📚 Read the Docs | 🐛 Report a Bug

✨ Features

• HTTP by default. Direct requests via curl_cffi in impersonate mode + real browser headers generation.
• Browser on demand. Offline render of an already received response (no repeated HTTP) and JS execution.
• Unified state. Two‑way transfer of cookies and localStorage between HTTP and the browser (storage_state ⇄ session).
• Async by design. Native asyncio for predictable concurrency.

🚀 Quick Start

Installation

pip install human-requests[playwright-stealth]
playwright install

Direct request (pretend to be a browser)

import asyncio
from human_requests import Session, HttpMethod

async def main():
    async with Session(headless=True, browser="camoufox") as s:
        resp = await s.request(HttpMethod.GET, "https://target.example/")
        print(resp.status_code, len(resp.text))

asyncio.run(main())

Render an already received response (without another request)

# resp — the result of an HTTP request
async with resp.render(wait_until="networkidle") as page:
    await page.wait_for_selector("#content")

# after exiting:
# - cookies and localStorage are synced back into the session

Warm‑up: inject localStorage BEFORE page start

origin = "https://target.example"

async with Session(headless=True, browser="camoufox") as s:
    # prepare storage_state in advance
    s.local_storage.setdefault(origin, {})
    s.local_storage[origin]["seen"] = "1"
    s.local_storage[origin]["ab_variant"] = "B"

    # the browser starts with the required values already in place
    async with s.goto_page(f"{origin}/", wait_until="networkidle"):
        pass

Accessing state

# Cookies:
print(s.cookies.storage)

# LocalStorage:
print(s.local_storage.get("https://target.example", {}))

Key Characteristics

• HTTP impersonation: curl_cffi + browser‑grade headers on every request.
• Offline render: first response interception (fulfill) and soft reloads without recreating contexts.
• State as a first‑class citizen: cookies and localStorage sync both ways.
• Unified proxy layer: single proxy format → for curl_cffi and Playwright.
• Clean stack: no external Go binaries.

Comparison: human-requests vs hrequests

Aspect	human-requests	hrequests
Execution model	asyncio (native)	sync + threads/gevent
HTTP impersonation	curl_cffi impersonate + per‑request browser headers	tls-client (Go backend)
Offline Response render	Yes (fulfill + soft‑reload; no repeated HTTP)	Yes (post‑render with cookies/content update)
Cookies ↔ HTTP/Browser	Two‑way transfer	Two‑way transfer
localStorage ↔ HTTP/Browser	First‑class (storage_state ⇄ session)	Via page.evaluate(...)
Typing	mypy‑friendly	—
Dependencies	No Go binaries	Go backend (tls-client)
Built‑in HTML parser	—	selectolax

The focus of human-requests is a controlled anti‑bot pipeline in asyncio: HTTP by default, a browser only where needed, with state hand‑off.

🛠️ Development

Setup

git clone https://github.com/Miskler/human-requests.git
cd human-requests
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
make build
make install-dev

Commands

# Checks
pytest          # tests + coverage
make lint       # ruff/flake8/isort/black (if enabled)
make type-check # mypy/pyright
# Actions
make format     # formatting
make docs       # build documentation

Dev: local test server

# from the test_server/ folder
make serve  # foreground (Ctrl+C to stop)
make stop   # stop background process