PageFetch

Async-first web scraping and content extraction library for Python 3.11+ — with intelligent browser fallback, built-in caching, SSRF protection, and structured Markdown output.

PageFetch is a modern, asynchronous Python library for fetching web pages and extracting clean, structured content. It attempts a pooled HTTP request first, scores the returned HTML using multiple completeness signals, and lazily falls back to a Camoufox stealth browser when the page looks blocked, empty, or dependent on client-side JavaScript rendering. The result is always a consistent FetchResult object carrying Markdown, raw HTML, extracted links, images, metadata, and more — regardless of which method succeeded.

Perfect for web scraping, content aggregation, LLM data pipelines, SEO analysis, archiving, and any workflow that needs reliable page content without fighting bot detection or leaking internal infrastructure.

Supported environments: Windows 10/11 x64 and Ubuntu 22.04/24.04 x64 on Python 3.11–3.13. Browser mode relies on the corresponding upstream Camoufox artifact.

Table of Contents

• Quick Start
• Why PageFetch
• Security & SSRF Protection
• Stealth & Anti-Detection
• How It Works
• Installation
• API Reference
• Output Model
• CLI Usage
• Caching
• Proxy Support
• Non-HTML Content
• Configuration Reference
• Development
• License

Quick Start

import asyncio

from pagefetch import PageFetch


async def main() -> None:
    async with PageFetch(mode="auto", cache_ttl="24h") as client:
        result = await client.fetch("https://example.com")
        if result.success:
            print(f"Title: {result.title}")
            print(f"Markdown length: {len(result.markdown or '')} chars")
            print(f"Links found: {len(result.links or [])}")
            print(f"Images found: {len(result.images or [])}")
        else:
            print(f"Failed: {result.error.message}")


asyncio.run(main())

Fetch Multiple URLs in Parallel

async with PageFetch(mode="auto") as client:
    results = await client.fetch_many([
        "https://example.com",
        "https://example.org",
        "https://httpbin.org/html",
    ])
    for r in results:
        status = "✓" if r.success else "✗"
        print(f"{status} {r.url} ({r.fetch_method}, {r.duration_ms:.0f}ms)")

Why PageFetch

Security & SSRF Protection

PageFetch incorporates strict Server-Side Request Forgery (SSRF) defense at both the HTTP and browser layers:

• Restricted Targets: Any attempt to fetch private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), loopback addresses (127.0.0.0/8, localhost), link-local IPs, multicast ranges, or cloud provider metadata endpoints (169.254.169.254) is rejected immediately with an invalid_url error.
• Redirect Validation: HTTP redirect chains validate each hop against SSRF rules, preventing open redirect bypasses.
• Browser Route Interception: Camoufox browser instances intercept page-level and subresource network routes to block access to restricted hosts and internal IP ranges.
• Opt-in Runtime Installs: Dynamic package installation at runtime is disabled by default (PAGEFETCH_AUTO_INSTALL=0) to prevent unexpected execution of external install scripts.

Stealth & Anti-Detection

When mode is "auto" or "browser", PageFetch exposes a layered stealth posture so the browser fingerprint can stay aligned with the proxy exit country:

# Quiet, fast default for open sites
async with PageFetch(mode="auto") as client:
    ...

# Maximum stealth for heavily protected targets
async with PageFetch(
    mode="browser",
    proxy="decodo",
    stealth_level="max",
    proxy_geo="DE",
) as client:
    result = await client.fetch("https://example.com")

The CLI exposes every knob via --stealth-level, --block-level, --humanize / --no-humanize, --session-rotation, --request-pacing, --accept-language, --proxy-geo, and --cleaning-level.

Platform-aware headless mode

Camoufox launches in a different mode per OS so the fingerprint stays realistic on every host:

• Linux — headless=False against an in-process Xvfb. The browser renders a real window into the virtual display, avoiding the fingerprinting tells that come with Firefox's --headless flag. Display lifecycles are strictly managed to avoid orphan processes.
• Windows / macOS — Firefox's native headless=True. No display server is required.

On Linux, Xvfb must be installed (e.g. apt install xvfb on Debian/Ubuntu, dnf install xorg-x11-server-Xvfb on Fedora/RHEL). When the binary is missing, browser fetches surface a FetchErrorInfo with code xvfb_missing.

How It Works

1. Normalize & Validate — URL syntax, encoding, scheme, and host safety (SSRF checks) are validated before any network call.
2. Check cache — If a valid SQLite entry exists, return instantly.
3. HTTP fetch — Performed via httpx with HTTP/2, connection pooling, and bounded retries. Non-HTML responses (PDF, XML, plain text) are processed directly without browser overhead.
4. Content analysis — The confidence score evaluates HTML completeness using text density, structural markup, heading presence, link counts, and common blocking signals (captcha walls, empty bodies, access-denied patterns).
5. Browser fallback (auto mode) — Camoufox takes over when HTTP content confidence is below the threshold (default 0.80), or when the server returns a blocked status (403 or 429). Blocked responses bypass unnecessary HTTP retries and switch directly to browser rendering. Timeouts, connection failures, 404s, and 5xx responses fail fast at the HTTP layer.
6. Processing pipeline — Cleaned HTML → extracted links, images, metadata → converted to Markdown via an engine preserving tables (with proper escaping of table pipes), code blocks, and nested lists.
7. Cache & return — The structured FetchResult is persisted to SQLite and returned.

Installation

Prerequisites

• Python ≥ 3.11
• Xvfb (Linux only, for stealth browser mode: sudo apt install xvfb)

Install

# Core HTTP/HTML support (lightweight)
pip install .

# Add browser fallback
pip install ".[browser]"
python -m camoufox fetch

# Add PDF extraction
pip install ".[pdf]"

# Or install every optional feature
pip install ".[all]"
python -m camoufox fetch

HTTP mode works without Camoufox. For browser fallback or browser mode, install pagefetch[browser] and fetch the browser artifact via python -m camoufox fetch. Automatic package and binary installation on first use is disabled by default for safety, but can be explicitly enabled with PAGEFETCH_AUTO_INSTALL=1.

API Reference

PageFetch Client

The main entry point. Use as an async context manager for automatic cleanup.

from pagefetch import PageFetch

client = PageFetch(
    mode="auto",              # "auto" | "http" | "browser"
    proxy="none",             # "none" | "decodo" | "dataimpulse"
    cleaning_level="standard", # "minimal" | "standard" | "maximum"
    http_concurrency=10,      # Max parallel HTTP requests
    browser_concurrency=4,    # Max parallel browser instances
    cache_enabled=True,       # Enable SQLite disk cache
    cache_ttl="24h",          # TTL: "30m", "2h", "7d", or seconds as int
    cache_path=None,          # Custom SQLite cache path (None = platform default)
    http_timeout=20.0,        # Per-request HTTP timeout (seconds)
    browser_timeout=45.0,     # Per-page browser timeout (seconds)
    retries_http=3,           # Retry on 5xx (and HTTP-only 429) errors
    retries_browser=2,        # Retry on browser failure
    max_redirects=10,         # Maximum redirect chain
    max_content_size=25 * 1024 * 1024,  # Max response body bytes (25 MiB)
    confidence_threshold=0.80,    # Min confidence before browser fallback
    block_images=True,        # Block image loading in browser mode to save bandwidth
    block_level="aggressive", # "minimal" | "balanced" | "aggressive"
    accept_language="en-US,en;q=0.5",  # Accept-Language header
    humanize=False,          # Add small randomized delays to mimic a human
    session_rotation="sticky",# "sticky" | "rotate" proxy session strategy
    request_pacing=0.0,       # Seconds of delay between requests
    stealth_level="off",      # "off" | "balanced" | "max" preset
    proxy_geo=None,           # ISO 3166-1 alpha-2 (e.g. "US", "DE")
    raise_on_error=False,     # Raise PageFetchError instead of returning error result
    screenshot_max_bytes=50 * 1024 * 1024,  # Max bytes for extract(screenshot=…)
)

Modes

Methods

fetch(url, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → FetchResult

Fetch a single URL. All keyword arguments override client-level defaults for this request. Bare domains (e.g. example.com) are automatically normalized to https://.

fetch_many(urls, *, mode=None, proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → list[FetchResult]

Fetch multiple URLs concurrently. Deduplicates identical inputs, preserves original ordering, and isolates failures across URLs.

extract(url, *, structure=True, compact_structure=False, screenshot="none", screenshot_format="png", proxy=None, use_cache=True, cache_ttl=None, raise_on_error=None) → FetchResult

Fetch a page in browser mode and return the rendered DOM, an optional structural summary, and/or a screenshot:

• result.html — Fully rendered HTML.
• result.structure — PageStructure summary ($O(N)$ extraction with verified unique_selector).
• result.screenshot / result.screenshot_format — Captured screenshot bytes (screenshot="viewport" or screenshot="full").

Screenshots are bounded by screenshot_max_bytes (default 50 MiB) and are not cached in SQLite.

Public API Exports

Output Model

Every fetch returns a FetchResult dataclass:

from dataclasses import dataclass
from datetime import datetime

@dataclass
class FetchResult:
    url: str                    # Normalized request URL
    final_url: str | None       # URL after all redirects
    status_code: int | None     # HTTP status code
    success: bool               # Did the fetch succeed?
    content_type: str | None    # e.g. "text/html", "application/pdf"
    encoding: str | None        # Detected charset
    title: str | None           # Page <title> or PDF title
    markdown: str | None        # Cleaned Markdown body
    html: str | None            # Raw HTML (excluded from JSON by default)
    text: str | None            # Plain-text body fallback
    metadata: dict              # OpenGraph, Twitter Cards, meta tags
    links: list[LinkInfo]       # All <a> tags with text, URL, rel
    images: list[ImageInfo]     # All <img> tags with url, alt, title
    structure: PageStructure | None  # DOM/stylesheet/script summary (when requested)
    screenshot: bytes | None         # PNG/JPEG screenshot bytes (when requested)
    screenshot_format: str | None    # "png" or "jpeg"
    fetch_method: str | None    # "http" or "browser"
    proxy_provider: str         # "none", "decodo", or "dataimpulse"
    content_confidence: float | None  # 0–1 completeness score (None for browser mode)
    from_cache: bool            # Was this served from cache?
    duration_ms: float | None   # Total fetch duration
    fetched_at: datetime | None # ISO 8601 timestamp
    warnings: list[str]         # Non-fatal issues (cache skip, etc.)
    error: FetchErrorInfo | None  # Error details when success=False

Serialization

# JSON output (HTML, structure, and screenshot excluded by default)
print(result.json(indent=2))
print(result.json(include_html=True))                    # Include raw HTML
print(result.json(include_structure=True))               # Include PageStructure summary
print(result.json(include_structure=True, compact_structure=True)) # Compact summary for LLMs
print(result.json(include_screenshot=True))              # Include base64-encoded screenshot

# Python dict
data = result.to_dict()
data = result.to_dict(include_screenshot=True)

# Reconstruct from dictionary
reconstructed = FetchResult.from_dict(data)

CLI Usage

PageFetch provides a powerful command-line interface accessible via pagefetch or python -m pagefetch:

# Fetch and print Markdown (bare domains like example.com are supported)
pagefetch example.com --format markdown

# Fetch raw HTML
pagefetch https://example.com --format html

# Output structured JSON
pagefetch https://example.com --format json

# Take a full-page screenshot and get JSON with base64 screenshot data
pagefetch https://example.com --format json --screenshot viewport

# Inspect page structure as Markdown
pagefetch https://example.com --format structure

# Raw page shell (HTML + structure + screenshot)
pagefetch https://example.com --format raw --screenshot full --screenshot-format png

# Save output to a file
pagefetch https://example.com --mode browser -o output.md

# Process multiple URLs from a file
pagefetch urls.txt --format json --mode auto

# Use residential proxy with German exit node
pagefetch https://example.com --proxy decodo --proxy-geo DE

# Apply stealth preset with session rotation
pagefetch https://example.com --mode browser --stealth-level balanced --session-rotation rotate

# Debug mode for troubleshooting
pagefetch https://example.com --debug

CLI Options

Exit codes: 0 all succeeded, 1 all failed, 2 usage/IO error, 3 partial failure.

Interactive Menu

Running python -m pagefetch with no arguments launches a guided interactive terminal menu (mode, proxy, stealth, format, URL entry). Supplying any argument (e.g. python -m pagefetch example.com) directly runs the CLI.

Caching

PageFetch uses a SQLite-backed disk cache stored in the platform's user cache directory (platformdirs). Cache keys incorporate the normalized URL, fetch mode, proxy configuration, and relevant stealth settings.

• Default TTL: 24 hours (configurable: "30m", "2h", "7d", or integer seconds).
• Network bypass: Cache hits return instantly without network or browser overhead.
• Graceful degradation: Cache read/write issues produce warnings without interrupting fetches.

# Bypass cache for a single request
result = await client.fetch("https://example.com", use_cache=False)

# Custom TTL per request
result = await client.fetch("https://example.com", cache_ttl="1h")

# Custom SQLite database location
client = PageFetch(cache_path="/path/to/custom_cache.sqlite3")

Proxy Support

PageFetch provides built-in integration with residential proxy providers:

# Use Decodo proxy with German exit node and aligned locale
async with PageFetch(proxy="decodo", proxy_geo="DE") as client:
    result = await client.fetch("https://example.de")

# Rotate proxy sessions across requests
async with PageFetch(proxy="dataimpulse", session_rotation="rotate") as client:
    results = await client.fetch_many([...])

Credentials are never exposed in result objects, logs, or cache keys.

Non-HTML Content

Non-HTML content is handled immediately at the HTTP layer, bypassing all browser dependencies.

Configuration Reference

Development

# Clone and set up
git clone https://github.com/neuronaline/pagefetch.git && cd pagefetch

# Install with test dependencies
pip install -e ".[test]"

# Run test suite
pytest

# Code formatting & linting
ruff check .

License

PageFetch is released under the MIT License.

_{Built with ❤️ for developers who need reliable, structured web content without fighting bot detection.}