human-requests 0.2.2

Playwright wrapper обеспечивающий необходимый функционал для человеко-подобного парсинга сайтов

Human Requests

Asynchronous Playwright wrappers for browser-like HTTP scenarios, controlled render flow, and API autotest integration.

Star us on GitHub | Read the Docs | Report a Bug

Features

• Typed wrappers over Playwright primitives:
  • HumanBrowser
  • HumanContext
  • HumanPage
• HumanPage.fetch(...): execute HTTP requests from page context and get structured FetchResponse.
• HumanPage.goto_render(...): render already available response payloads without duplicate upstream request.
• Storage helpers:
  • HumanContext.local_storage() for full context snapshot
  • HumanPage.local_storage() for current page origin
  • HumanPage.cookies() convenience alias
• Fingerprint snapshot collection: HumanContext.fingerprint(...).
• Built-in pytest autotest plugin for API clients (@autotest, hooks, params, dependencies).

Installation

Base package:

pip install human-requests
playwright install chromium

Optional autotest addon dependencies:

pip install human-requests[autotest] pytest pytest-anyio pytest-jsonschema-snapshot pytest-subtests

If you run with Camoufox, install it separately:

pip install camoufox
camoufox fetch

Quick Start

Wrap a Playwright browser

import asyncio
from playwright.async_api import async_playwright

from human_requests import HumanBrowser


async def main() -> None:
    async with async_playwright() as p:
        pw_browser = await p.chromium.launch(headless=True)
        browser = HumanBrowser.replace(pw_browser)

        ctx = await browser.new_context()
        page = await ctx.new_page()

        await page.goto("https://httpbin.org/html", wait_until="domcontentloaded")
        print(page.url)

        await browser.close()


asyncio.run(main())

Direct request in page context (fetch)

resp = await page.fetch("https://httpbin.org/json")
print(resp.status_code)
print(resp.json())

Render previously fetched response (goto_render)

challenge = await page.fetch("https://example.com/challenge")
await page.goto_render(challenge, wait_until="networkidle")

Auto Screenshot on Handler Failure

If you want a screenshot to be taken when errors like playwright.Error (including timeouts) occur, you can set page.on_error_screenshot_path = "screenshot.png" (this setting is page-specific; by default, screenshots are disabled).

State helpers

cookies = await page.cookies()
context_storage = await ctx.local_storage()
page_storage = await page.local_storage()
print(len(cookies), context_storage.keys(), page_storage.keys())

Fingerprint snapshot

fingerprint = await ctx.fingerprint(origin="https://example.com")
print(fingerprint.user_agent)
print(fingerprint.browser_name, fingerprint.browser_version)

API Autotest Addon (pytest)

human-requests ships with a pytest plugin that can auto-run API methods marked with @autotest and validate payloads via schemashot from pytest-jsonschema-snapshot. With pytest-subtests, each discovered @autotest method and @autotest_data provider is shown as a separate subtest inside test_autotest_api_methods.

Minimal pytest.ini:

[pytest]
anyio_mode = auto
autotest_start_class = your_package.StartClass
autotest_typecheck = warn
autotest_trace_limit = 3
autotest_truncation_context_lines = 3

autotest_typecheck modes:

• off (default): no runtime type checks for params provider arguments
• warn: emit RuntimeWarning on annotation mismatch
• strict: fail test case with TypeError on mismatch

autotest_trace_limit controls how many Source: blocks are kept from the end of a crash report call chain. The default is 3; set it to 0 to hide the source chain entirely.

autotest_truncation_context_lines controls how many source context lines are kept on each side of a split crash-report excerpt before [log content truncated]. The function header itself is always shown. The default is 3.

Minimal fixtures:

import pytest
from your_package import StartClass


@pytest.fixture(scope="session")
def anyio_backend() -> str:
    return "asyncio"


@pytest.fixture(scope="session")
async def api() -> StartClass:
    async with StartClass() as client:
        yield client

Business code only marks methods:

from human_requests import autotest


class Catalog:
    @autotest
    async def tree(self):
        ...

Test layer adds hooks and params:

from human_requests import autotest_depends_on, autotest_hook, autotest_params
from human_requests.autotest import AutotestCallContext, AutotestContext


@autotest_hook(target=Catalog.tree)
def _capture_category(_resp, data, ctx: AutotestContext) -> None:
    ctx.state["category_id"] = data["items"][0]["id"]


@autotest_depends_on(Catalog.tree)
@autotest_params(target=Catalog.feed)
def _feed_params(ctx: AutotestCallContext) -> dict[str, int]:
    return {"category_id": ctx.state["category_id"]}

Parent-specific registration is supported:

@autotest_hook(target=Child.method, parent=ParentA)
def _only_for_parent_a(_resp, data, ctx):
    ...

For a complete guide, see docs/source/autotest.rst.

Development

Setup:

git clone https://github.com/Miskler/human-requests.git
cd human-requests
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .

Commands:

pytest
make lint
make type-check
make format
make docs