PageFetch

Async-first web scraping and content extraction library for Python 3.11+ — with intelligent browser fallback, built-in caching, and structured Markdown output.

PageFetch is a modern, asynchronous Python library for fetching web pages and extracting clean, structured content. It attempts a pooled HTTP request first, scores the returned HTML using multiple completeness signals, and lazily falls back to a Camoufox stealth browser when the page looks blocked, empty, or dependent on client-side JavaScript rendering. The result is always a consistent FetchResult object carrying Markdown, raw HTML, extracted links, images, metadata, and more — regardless of which method succeeded.

Perfect for web scraping, content aggregation, LLM data pipelines, SEO analysis, archiving, and any workflow that needs reliable page content without fighting bot detection.

Supported environments: Windows 10/11 x64 and Ubuntu 22.04/24.04 x64 on Python 3.11–3.13. Browser mode relies on the corresponding upstream Camoufox artifact.

Table of Contents

• Quick Start
• Why PageFetch
• Stealth & Anti-Detection
• How It Works
• Installation
• API Reference
• Output Model
• CLI Usage
• Caching
• Proxy Support
• Non-HTML Content
• Configuration Reference
• Development
• License

Quick Start

import asyncio

from pagefetch import PageFetch


async def main() -> None:
    async with PageFetch(mode="auto", cache_ttl="24h") as client:
        result = await client.fetch("https://example.com")
        if result.success:
            print(f"Title: {result.title}")
            print(f"Markdown length: {len(result.markdown or '')} chars")
            print(f"Links found: {len(result.links or [])}")
            print(f"Images found: {len(result.images or [])}")
        else:
            print(f"Failed: {result.error.message}")


asyncio.run(main())

Fetch Multiple URLs in Parallel

async with PageFetch(mode="auto") as client:
    results = await client.fetch_many([
        "https://example.com",
        "https://example.org",
        "https://httpbin.org/html",
    ])
    for r in results:
        status = "✓" if r.success else "✗"
        print(f"{status} {r.url} ({r.fetch_method}, {r.duration_ms:.0f}ms)")

Why PageFetch

Stealth & Anti-Detection

When mode is "auto" or "browser", PageFetch exposes a layered stealth posture so the browser fingerprint can stay aligned with the proxy exit country:

# Quiet, fast default for open sites
async with PageFetch(mode="auto") as client:
    ...

# Maximum stealth for heavily protected targets
async with PageFetch(
    mode="browser",
    proxy="decodo",
    stealth_level="max",
    proxy_geo="DE",
) as client:
    result = await client.fetch("https://example.com")

The CLI exposes every knob via --stealth-level, --block-level, --humanize / --no-humanize, --session-rotation, --request-pacing, --accept-language, --proxy-geo, and --cleaning-level.

Platform-aware headless mode

Camoufox launches in a different mode per OS so the fingerprint stays realistic on every host:

• Linux — headless=False against an in-process Xvfb. The browser renders a real window into the virtual display, avoiding the fingerprinting tells that come with Firefox's --headless flag.
• Windows / macOS — Firefox's native headless=True. No display server is required.

On Linux, Xvfb must be installed (e.g. apt install xvfb on Debian/Ubuntu, dnf install xorg-x11-server-Xvfb on Fedora/RHEL). When the binary is missing, browser fetches surface a FetchErrorInfo with code xvfb_missing.

How It Works

1. Normalize the URL (scheme, encoding, fragments).
2. Check cache — if a valid SQLite entry exists, return instantly.
3. HTTP fetch using httpx with HTTP/2, connection pooling, and configurable retries.
   Non-HTML responses (PDF, XML, plain text) are handled directly without browser overhead.
4. Content analysis — the confidence score evaluates HTML completeness using
   text density, structural markup, heading presence, link counts, and common blocking signals
   (captcha walls, empty bodies, access-denied patterns).
5. Browser fallback (auto mode only) — Camoufox takes over only when HTTP content
   confidence is below the threshold (default 0.80), or when the server returns a blocked
   status (403/429). Timeouts, connection failures, 404s, and 5xx responses fail fast at
   the HTTP layer instead of waiting on a browser navigation.
6. Processing pipeline — cleaned HTML → extracted links, images, metadata →
   converted to Markdown via a custom converter that preserves tables, code blocks,
   and nested lists.
7. Cache & return — the structured FetchResult is persisted to SQLite and returned.

Installation

Prerequisites

• Python ≥ 3.11

Install

# Core HTTP/HTML support
pip install .

# Add browser fallback
pip install ".[browser]"
python -m camoufox fetch

# Add PDF extraction
pip install ".[pdf]"

# Or install every optional feature
pip install ".[all]"

HTTP mode works without Camoufox. On the first use of browser fallback or browser mode, PageFetch installs the optional browser package and runtime automatically (unless PAGEFETCH_AUTO_INSTALL=0 is set). The commands above remain available when you want to provision the browser feature explicitly.

API Reference

PageFetch Client

The main entry point. Use as an async context manager for automatic cleanup.

from pagefetch import PageFetch

client = PageFetch(
    mode="auto",              # "auto" | "http" | "browser"
    proxy="none",             # "none" | "decodo" | "dataimpulse"
    cleaning_level="standard", # "minimal" | "standard" | "maximum" — how aggressively to strip non-content DOM
    http_concurrency=10,      # Max parallel HTTP requests
    browser_concurrency=4,    # Max parallel browser instances
    cache_enabled=True,       # Enable SQLite disk cache
    cache_ttl="24h",          # TTL: "30m", "2h", "7d", or seconds as int
    cache_path=None,          # Custom SQLite cache path (None = platform default)
    http_timeout=20.0,        # Per-request HTTP timeout (seconds)
    browser_timeout=45.0,     # Per-page browser timeout (seconds)
    retries_http=3,           # Retry on 429/5xx for HTTP
    retries_browser=2,        # Retry on browser failure
    max_redirects=10,         # Maximum redirect chain
    max_content_size=25 * 1024 * 1024,  # Max response body bytes (25 MiB)
    confidence_threshold=0.80,    # Min confidence before browser fallback
    block_images=True,        # Block image loading in browser mode to save bandwidth
    block_level="aggressive", # "minimal" | "balanced" | "aggressive" (ignored when stealth_level != "off")
    accept_language="en-US,en;q=0.5",  # Accept-Language header
    humanize=False,          # Add small randomized delays to mimic a human
    session_rotation="sticky",# "sticky" | "rotate" proxy session strategy
    request_pacing=0.0,       # Seconds of delay between requests
    stealth_level="off",      # "off" | "balanced" | "max" preset (sets humanize, block_level, pacing, session_rotation)
    proxy_geo=None,           # ISO 3166-1 alpha-2 (e.g. "US", "DE") to align locale + Accept-Language
    raise_on_error=False,     # Raise PageFetchError instead of returning error result
    screenshot_max_bytes=50 * 1024 * 1024,  # Max bytes for extract(screenshot=…); oversized captures are dropped
)

Modes

Methods

fetch(url, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → FetchResult

Fetch a single URL. All keyword arguments override the client-level defaults for this individual request only. For structural / RAW HTML / screenshot capture use extract() instead — it always runs in browser mode and returns the rendered page shell.

fetch_many(urls, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → list[FetchResult]

Fetch multiple URLs concurrently. Deduplicates identical inputs internally, preserves the original input order, and isolates individual failures — one bad URL never affects the others.

extract(url, *, structure=True, compact_structure=False, screenshot="none", screenshot_format="png", proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → FetchResult

Fetch a page and return the rendered DOM plus, on demand, a structural summary and/or a screenshot. extract() always uses mode="browser" — there is no HTTP→browser pipeline because the goal is the full page shell, not the article-shaped extraction. Returns a FetchResult populated with:

• result.html — the browser-rendered HTML.
• result.structure — PageStructure summary (when structure=True).
• result.screenshot / result.screenshot_format — captured screenshot bytes (when screenshot != "none"); screenshot="viewport" captures the visible area, screenshot="full" captures the entire scrollable page, encoded as screenshot_format (png or jpeg).

Screenshots are bounded by PageFetchConfig.screenshot_max_bytes (default 50 MiB) — oversized captures are discarded with a warning. Screenshots are not persisted in the SQLite cache; when a cached result is returned and a screenshot was requested, result.warnings carries a hint to re-fetch with use_cache=False.

Public API Exports

Everything PageFetch ships in its top-level pagefetch namespace:

Browser dependencies are auto-installed on first browser use. Call ensure_runtime_requirements() for an up-front check without installing, or auto_bootstrap_browser() to force installation at any point.

Set PAGEFETCH_AUTO_INSTALL=0 (also accepts false, no, or off) to disable automatic browser installation. In that case, install pagefetch[browser] and run python -m camoufox fetch before browser use.

Output Model

Every fetch returns a FetchResult dataclass:

from dataclasses import dataclass

@dataclass
class FetchResult:
    url: str                    # Normalized request URL
    final_url: str | None       # URL after all redirects
    status_code: int | None     # HTTP status code
    success: bool               # Did the fetch succeed?
    content_type: str | None    # e.g. "text/html", "application/pdf"
    encoding: str | None        # Detected charset
    title: str | None           # Page <title> or PDF title
    markdown: str | None        # Cleaned Markdown body
    html: str | None            # Raw HTML (excluded from JSON by default)
    text: str | None            # Plain-text body fallback
    metadata: dict              # OpenGraph, Twitter Cards, meta tags
    links: list[LinkInfo]       # All <a> tags with text, URL, rel
    images: list[ImageInfo]     # All <img> tags with url, alt, title
    structure: PageStructure | None  # Bounded DOM/stylesheet/script summary (only when requested)
    screenshot: bytes | None         # PNG/JPEG screenshot bytes (only when requested)
    screenshot_format: str | None    # "png" or "jpeg"
    fetch_method: str | None    # "http" or "browser"
    proxy_provider: str         # "none", "decodo", or "dataimpulse"
    content_confidence: float | None  # 0–1 completeness score (None for browser mode)
    from_cache: bool            # Was this served from cache?
    duration_ms: float | None   # Total fetch duration
    fetched_at: datetime | None # ISO 8601 timestamp
    warnings: list[str]         # Non-fatal issues (cache skip, etc.)
    error: FetchErrorInfo | None  # Error details when success=False

Serialization

# JSON output (HTML, structure, and screenshot excluded by default for compactness)
print(result.json(indent=2))
print(result.json(include_html=True))                    # Include raw HTML
print(result.json(include_structure=True))               # Include PageStructure summary
print(result.json(include_structure=True,                # Trimmed structure payload for LLM consumers
                  compact_structure=True))
print(result.json(include_screenshot=True))              # Include base64-encoded screenshot

# Python dict
data = result.to_dict()
data = result.to_dict(include_html=True)
data = result.to_dict(include_structure=True)
data = result.to_dict(include_structure=True, compact_structure=True)
data = result.to_dict(include_screenshot=True)

# Reconstruct from cached JSON
reconstructed = FetchResult.from_dict(data)

Note: compact_structure=True is inspection-only — inline <style> / <script> previews are returned as {length, preview} and the lossy payload cannot be reconstructed with FetchResult.from_dict.

Extraction (RAW HTML, Structure, Screenshot)

When you want the full rendered page shell — RAW HTML plus the optional structural summary and/or screenshot — call extract() instead of fetch(). extract() always uses browser mode (no auto fallback) and returns a FetchResult with the rendered DOM:

async with PageFetch() as client:
    # RAW HTML + structural summary (no screenshot)
    result = await client.extract("https://example.com")
    print(result.html)
    print(result.structure.root.selector)

    # Capture a full-page PNG screenshot too
    captured = await client.extract(
        "https://example.com",
        screenshot="full",
        screenshot_format="png",
    )
    with open("page.png", "wb") as fh:
        fh.write(captured.screenshot)

extract() runs the same readiness / scroll pipeline as a browser-mode fetch(), so the captured HTML matches what a visitor sees. Screenshots follow Playwright's page.screenshot() semantics — screenshot="viewport" captures the initial visible area, screenshot="full" captures the entire scrollable page. Use --screenshot-format=jpeg (or screenshot_format="jpeg" in Python) to compress full-page captures.

The structural summary produced by extract() includes:

• A nested DOM tree with filtered attributes, short direct-text previews, a compact selector, a deterministic CSS path, and a verified unique_selector suitable for starting scraper rules. It remains bounded by max_depth and max_nodes so the payload stays predictable even on enormous pages.
• External stylesheet URLs (<link rel="stylesheet">) with media, integrity, and crossorigin hints.
• Inline <style> blocks with a per-block preview and a truncated flag.
• External script URLs (<script src="…">) with type, async, defer, integrity, and crossorigin.
• Inline <script> blocks with a per-block preview, type, and a truncated flag.

Use the lower-level helper directly when you already have parsed HTML:

from pagefetch import StructureLimits, extract_structure

structure = extract_structure(html, base_url="https://example.com/")

StructureLimits exposes max_depth, max_nodes, text_preview, and inline_source_limit for callers that need different safety bounds.

CLI Usage

PageFetch ships with a command-line interface accessible via pagefetch:

# Fetch and print Markdown
pagefetch https://example.com --format markdown

# Fetch and print raw HTML
pagefetch https://example.com --format html

# Fetch from a list and output JSON
pagefetch urls.txt --format json --mode auto

# Save output to a file
pagefetch https://example.com --mode browser -o output.md

# Structured JSON with raw HTML included
pagefetch https://example.com --format json --include-html

# Inspect the page structure as Markdown
pagefetch https://example.com --format structure

# Raw page shell: HTML + structure + (optionally) screenshot, single JSON doc
pagefetch https://example.com --format raw --screenshot full --screenshot-format png

# Include a PageStructure summary inside the regular JSON output
pagefetch https://example.com --format json --include-html

# Multiple URLs from a file (one URL per line)
pagefetch urls.txt --format json --mode auto

# Load configuration from a YAML file with CLI overrides
pagefetch --config config.yaml --mode browser https://example.com

# Override cache TTL and disable image loading
pagefetch https://example.com --cache-ttl 1h --no-block-images

# Use the Decodo proxy with a German exit and locale alignment
pagefetch https://example.com --proxy decodo --proxy-geo DE

# Apply a balanced stealth preset with a rotated proxy session
pagefetch https://example.com --mode browser --stealth-level balanced --session-rotation rotate

# Verbose logging for debugging
pagefetch https://example.com --debug

CLI arguments map directly to the Python API:

The --screenshot flag (and --format raw) auto-promote the request to browser mode and route through PageFetch.extract().

Exit codes: 0 all succeeded, 1 all failed, 2 usage/IO error, 3 partial failure.

Interactive Menu

Running python -m pagefetch with no arguments opens a guided interactive menu (mode, proxy, stealth, format, URL/file input, …) — useful when you don't want to memorise the flags. Pass python -m pagefetch --cli (or invoke the pagefetch console script) to use the argparse CLI documented above.

Caching

PageFetch uses a SQLite-backed disk cache (platformdirs user cache directory by default). Cache entries are keyed by normalized URL + mode + proxy + relevant fetch settings, so switching from "auto" to "browser" mode produces a different cache key.

• Default TTL: 24 hours (configurable: "30m", "2h", "7d", or integer seconds)
• Automatic: cache hits skip all network and browser work
• Graceful degradation: cache read/write failures never crash a fetch — they produce warnings

# Disable caching for a single request
result = await client.fetch("https://example.com", use_cache=False)

# Override TTL per-request
result = await client.fetch("https://example.com", cache_ttl="1h")

# Use a custom cache location
client = PageFetch(cache_path="/path/to/custom_cache.sqlite3")

Proxy Support

PageFetch natively supports rotating residential proxy providers:

# Use a proxy provider
async with PageFetch(proxy="decodo") as client:
    result = await client.fetch("https://example.com")

# Align locale + Accept-Language with the proxy exit country
async with PageFetch(proxy="decodo", proxy_geo="DE") as client:
    result = await client.fetch("https://example.de")

# Force a fresh proxy session per request
async with PageFetch(proxy="dataimpulse", session_rotation="rotate") as client:
    results = await client.fetch_many([...])

Credentials are never included in results, logs, or cache keys. Configure either a full proxy URL or the individual components — PageFetch validates both forms automatically. proxy_geo requires one of the countries defined in PageFetch's GEO_MAP (case-insensitive ISO 3166-1 alpha-2).

Non-HTML Content

PageFetch handles content types beyond HTML natively:

No browser overhead is incurred for non-HTML content — detection happens at the HTTP response level before any processing pipeline runs.

Configuration Reference

Development

# Clone and set up
git clone <repo-url> && cd pagefetch

# Install with test dependencies
pip install -e ".[test]"

# Run the test suite (browser integration tests are opt-in)
pytest

# Lint
ruff check .

Browser integration requires the separately downloaded Camoufox binary and is therefore kept optional in deterministic test environments. The test suite is designed to run fully offline — URLs are served via local fixtures.

License

PageFetch is released under the MIT License.

_{Built with ❤️ for developers who need reliable, structured web content without fighting bot detection.}