web-fetch-mcp

A Python MCP server using browser fingerprint impersonation, headless Chrome, and multi-tier anti-detection to reliably fetch web pages for AI agents — published on PyPI with CI/CD pipeline and test coverage.

What is this?

web-fetch-mcp is an MCP server that lets AI assistants (like Claude, Cursor, or any MCP-compatible client) fetch web pages reliably.

The problem it solves

When an AI agent fetches a web page, many sites return a CAPTCHA, a JavaScript challenge, or a login wall — but still send HTTP status 200 OK. A naive fetcher hands this garbage to the AI, which then reasons from nonsense.

How this tool fixes it

web-fetch-mcp detects when a site returns a block page instead of real content, and either:

1. Escalates to a stronger fetching strategy (there are 3 tiers), or
2. Fails loudly with a clear error (FetchBlocked) — never silently returning junk.

Status: Alpha. Core logic is tested, but real-world bypass benchmarks are in progress.

Installation

Requirements: Python 3.11+

pip install web-fetch-mcp

That's it. This installs the web-fetch-mcp command on your system.

Quick Start

1. Run the server

web-fetch-mcp

This starts the MCP server in the background. It doesn't show anything on its own — your AI client talks to it automatically.

2. Register with your MCP client

Add this to your MCP client's configuration (e.g. claude_desktop_config.json, .cursor/mcp.json, or equivalent):

{
  "mcpServers": {
    "web-fetch": {
      "command": "web-fetch-mcp"
    }
  }
}

3. Use it

Once connected, your AI assistant gains two tools:

Examples your AI can call:

# Get a clean article (strips navigation, ads, etc.)
fetch("https://example.com/blog-post", output="article")

# Get raw JSON from an API
fetch("https://api.github.com/repos/owner/repo")

# Force JavaScript rendering (for single-page apps like React/Vue)
fetch("https://spa-app.example.com", mode="dynamic")

# Take a screenshot of a dashboard
screenshot("https://example.com/dashboard")

How It Works

The server uses a 3-tier escalation ladder — it starts cheap and fast, only using expensive browser-based methods when simpler ones get blocked:

Tier 1: curl_cffi        — Fast static fetch (~500ms)
   ↓ (blocked?)
Tier 2: Patchright       — Real Chrome browser, renders JavaScript (~1-3s)
   ↓ (blocked?)
Tier 3: nodriver         — Stealth Chrome, evades automation detection (~2-4s)
   ↓ (still blocked?)
Raise FetchBlocked error — never return garbage to the AI

What each tier handles

Modes

You can control which tiers are used:

Output formats

Non-HTML content is auto-detected: JSON gets pretty-printed, PDFs get text-extracted.

Development Setup

If you want to contribute or modify the code:

git clone https://github.com/Dutta-SD/web-fetch-mcp.git
cd web-fetch-mcp
uv sync                    # install dependencies
uv pip install -e .        # install in editable mode
web-fetch-mcp              # run the server

Run tests:

uv run pytest

Project structure

src/web_fetch_mcp/
├── controller/    → Tool definitions (what the AI client can call)
├── service/       → Retry logic and escalation chain
├── accessor/      → Browser engines (the actual fetching code)
└── core/          → Shared utilities (config, block detection, rendering)

Responsible Use

This tool is for fetching content you are authorized to access. You are responsible for complying with each site's Terms of Service, robots.txt, and applicable law. The tool honors Retry-After headers and backs off by default. It does not solve CAPTCHAs or bypass authentication you don't hold.

Links

• PyPI: https://pypi.org/project/web-fetch-mcp/
• Source: https://github.com/Dutta-SD/web-fetch-mcp
• Issues: https://github.com/Dutta-SD/web-fetch-mcp/issues

Glossary

License

Apache-2.0