wildweb-mcp

An extensible Model Context Protocol (MCP) server, written in Python, that exposes web scraping as an MCP tool. It fetches a web page with a headless Chromium browser (so client-rendered pages are captured), strips boilerplate (navigation, ads, scripts, styles) with trafilatura, and returns token-efficient clean markdown to MCP clients. No API key or third-party search/scrape API is required — it talks to the open web directly.

Requirements

• Python 3.10+
• Chromium browser binary — installed automatically on first run, see below

Installation

Install into a virtualenv rather than system/global Python — it keeps wildweb-mcp's pinned dependencies isolated, and (as covered below) MCP clients need this venv's path anyway since they launch the server directly, bypassing your shell:

python3 -m venv ~/.venvs/wildweb
~/.venvs/wildweb/bin/pip install wildweb-mcp

(.venvs\wildweb\Scripts\pip.exe install wildweb-mcp on Windows.)

Upgrading

Using the same venv, add --upgrade:

~/.venvs/wildweb/bin/pip install --upgrade wildweb-mcp

Confirm the new version:

~/.venvs/wildweb/bin/python -c "import wildweb; print(wildweb.__version__)"

No MCP client config changes needed — the command path stays the same, it's the same venv with newer code inside it. Restart whichever MCP client has it running (Claude Desktop, Claude Code, etc.) so it relaunches the server process with the update.

The Chromium browser

pip install installs the playwright Python package (it's a pinned dependency of wildweb-mcp) but not the browser binary that Playwright drives.

You don't need to install it yourself. If Chromium isn't found when the server starts, it runs playwright install chromium automatically — a one-time download (~300MB) into the shared Playwright cache (~/.cache/ms-playwright on Linux/macOS, %USERPROFILE%\AppData\Local\ms-playwright on Windows) — then retries. Every startup after that is unaffected, since the binary is already there. This first-run install can take a few minutes and needs network access; if it can't finish within 5 minutes (e.g. no network) or fails for another reason, the server falls back to the existing clear startup error instead of hanging. Progress and any install errors are logged — see Where logs go below.

If you'd rather install it ahead of time (e.g. to avoid the first-run delay, or in an offline/ sandboxed environment where the server shouldn't attempt a network call on its own), run it yourself before starting the server — using the venv path from Installation above:

~/.venvs/wildweb/bin/playwright install chromium

(~/.venvs/wildweb/Scripts/playwright.exe install chromium on Windows.)

A bare playwright install chromium will not work as a copy-paste pre-setup step unless you first activate that venv (source ~/.venvs/wildweb/bin/activate, or on Windows ~/.venvs/wildweb/Scripts/activate) — without either activation or the full path, your shell has no playwright command on PATH at all (it only exists inside the venv you installed wildweb-mcp into) and the command fails with "command not found." The full-path form above works unconditionally, with nothing to activate first.

This has to be the playwright from the same environment wildweb-mcp was installed into — if you installed globally/user-wide instead (pip install --user or system pip) rather than into a venv, it lands in pip's script directory instead (e.g. ~/.local/bin on Linux, your Python install's Scripts/bin folder on macOS/Windows), and a bare playwright install chromium works directly only if that directory is on PATH.

On Linux, Chromium may also need OS-level shared libraries. This is never installed automatically — it requires root/sudo to run a system package manager, which the server will not do on its own. If the server's startup error (or a manual playwright install chromium) warns about missing dependencies, install them yourself:

playwright install-deps chromium

Configuration

The server reads a single JSON configuration file. Copy the sample and edit it as needed:

cp config.example.json config.json

{
  "request_timeout_ms": 30000,
  "token_budget": 4000,
  "log_level": "INFO",
  "continuation_ttl_seconds": 300,
  "browser": {
    "headless": true,
    "navigation_timeout_ms": 30000,
    "wait_until": "domcontentloaded",
    "max_retries": 2,
    "retry_backoff_ms": 500,
    "settle_ms": 1000
  }
}

All fields are optional and fall back to documented defaults:

Field	Default	Description
request_timeout_ms	30000	Logical request timeout in milliseconds.
token_budget	null	Max tokens per tool output; null disables budgeting/truncation.
log_level	"INFO"	One of DEBUG, INFO, WARNING, ERROR, CRITICAL.
continuation_ttl_seconds	300	Time-to-live for continuation tokens, in seconds.
browser.headless	true	Launch Chromium headless.
browser.navigation_timeout_ms	30000	Playwright navigation timeout; defaults to request_timeout_ms.
browser.wait_until	"domcontentloaded"	One of load, domcontentloaded, networkidle, commit. load waits for every subresource and can hang indefinitely on sites with a slow/stuck tracker or ad script.
browser.max_retries	2	Extra attempts after a transient navigation timeout or network error (connection reset, DNS hiccup); a non-2xx HTTP status is never retried. 0 disables retry.
browser.retry_backoff_ms	500	Base delay before each retry, doubled per attempt (500ms, then 1000ms, ...).
browser.settle_ms	1000	Extra wait after navigation completes, before reading the page, so content a site renders asynchronously after load has a chance to appear. Many modern (React/Next.js-style) sites return no content at all at 0 — set to 0 to disable if you don't need it.
search_result_count	10	Number of results the search tool returns per query.
search_timeout_ms	10000	Timeout in milliseconds for a search request.
user_agents	null	Array of User-Agent strings to rotate through; null uses the built-in default pool.
log_file	null	Path to also write logs to, capped at 10MB (see Where logs go); null logs to stderr only.

Simulating a modern browser

Every outgoing request (Playwright page navigation and the search tool's DuckDuckGo request) identifies as an ordinary modern desktop browser rather than a generic client/library default, since some sites are more likely to serve their normal response to a request that looks like one. Requests draw from a rotating pool of real browser User-Agent strings (Chrome, Firefox, Safari, Edge) round-robin, one per request, rather than reusing a single fixed value.

Set user_agents to replace the default pool with your own:

{
  "user_agents": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
  ]
}

The array must be non-empty; omit the field or set it to null to use the default pool.

Where logs go

All server logging — startup messages (including the automatic Chromium install above), per-request success/failure entries, and errors — is written to stderr, never stdout. This is deliberate: stdout is reserved exclusively for the MCP JSON-RPC protocol stream between the server and its client, so anything else written there would corrupt that stream.

What you see depends on how your MCP client surfaces a subprocess's stderr. Most already capture it to their own log file, so check there first:

• Claude Desktop: ~/Library/Logs/Claude/mcp-server-<name>.log (macOS), %APPDATA%\Claude\logs\mcp-server-<name>.log (Windows)
• Claude Code: claude mcp logs <name>
• Kiro CLI / opencode: each client's own MCP connection log/output panel — check its docs for the exact location

Or run wildweb directly in a terminal to see log lines interleaved with your own shell. log_level (see the Configuration table above) controls verbosity: DEBUG through CRITICAL, defaulting to INFO.

Writing logs to a file too

If your client doesn't capture stderr usefully, or you just want one log location regardless of which client launched the server, set log_file to a path:

{
  "log_file": "/absolute/path/to/wildweb.log"
}

This writes the same entries to that file in addition to stderr — nothing stops going to stderr. The file is capped at 10MB: once it hits that size, it's truncated (overwritten from empty) rather than rotated to .1/.2 backups, so it's always exactly one bounded file, never growing unbounded across a long-running server process and never leaving old backup files to clean up. If the path can't be opened (e.g. its parent directory doesn't exist), the server logs a warning and continues with stderr-only logging rather than failing to start.

Usage

A config file is optional. The config file path is resolved with the following precedence:

1. --config <path> command-line argument (highest precedence)
2. WILDWEB_CONFIG environment variable
3. If neither is provided, the server runs with the built-in defaults listed under Configuration — no config file is required.

# with no config at all — runs with built-in defaults
wildweb

# via command-line argument
wildweb --config config.json

# or via environment variable
WILDWEB_CONFIG=config.json wildweb

MCP clients connect over stdio and discover three tools:

• scrape — fetch a web page by URL and return its main textual content as clean markdown. When the content exceeds the configured token_budget, the result is truncated and a continuation token is returned. On a directory/listing page (a customer-stories grid, a blog index) where boilerplate removal would otherwise collapse the result down to just the one or two entries that look the most "article-like", a detected repeated group of same-class links is appended as a ## Links on this page section instead of being silently dropped.
• scrape_continue — fetch the next portion of a previously scraped page using a continuation token returned by scrape.
• search — search the web for a query and return a list of results (title, URL, snippet). Input: a non-empty text query. Output: up to search_result_count ordered results, or an error. The search tool does not fetch or scrape any result's page content — pipe a result's url into scrape for that.

Search provider caveat

The search tool is backed by a free, key-free SearchProvider implementation (DuckDuckGoSearchProvider) that fetches and parses DuckDuckGo's public HTML results page (https://html.duckduckgo.com/html/). This is an unofficial surface, not a documented or supported API:

• DuckDuckGo can change its HTML markup, rate-limit, or block automated requests at any time without notice, which will surface as a search_unavailable error from the search tool.
• Using it at any significant volume may be subject to DuckDuckGo's Terms of Service; this provider is intended for light, personal-scale use.
• The SearchProvider interface (src/wildweb/search/base.py) exists specifically so a different, officially-supported provider (e.g. a paid search API) can be registered later without changing the search tool or server wiring — see src/wildweb/search/registry.py.

Extraction limitations: repeated data widgets

trafilatura's boilerplate-removal is built around single articles (blog posts, news stories), and works by scoring out content that looks structurally like a nav menu, card grid, or sidebar: many short, repeated sibling elements. That heuristic can't distinguish a real repeated-link nav menu from a repeated data widget — a row of short "icon + value" tiles (a daily forecast strip's high/low temperatures, a stat dashboard's per-metric tiles, etc.) — since both look the same structurally (a list of many short, similarly-marked-up children). The result: that widget's actual values can be silently dropped from scrape's output even though the surrounding prose (summaries, descriptions) comes through fine.

This was observed concretely against a weather site's forecast page: the written summary, wind, and rain figures extracted correctly, but the high/low temperature tiles (each a short <li>-per-day value) did not, across every tuning option trafilatura exposes (favor_recall, custom extraction-size thresholds, the library's own more-liberal baseline() fallback) — this is an architectural limitation of generic article-extraction heuristics against dashboard-style UI, not something a config value fixes. scrape's existing repeated-links workaround (the ## Links on this page fallback described above) does not apply here either, since these widgets aren't <a> elements.

If you hit this on a specific site, there's no current config knob — the only generic recourse is confirming (in the raw page, e.g. via browser devtools) whether the missing data is actually present in the rendered DOM at all (if not, see browser.settle_ms above) and, if it's a widget-extraction gap rather than a missing-render gap, treating that value as currently unreachable via scrape on that page.

Adding it to an MCP client

Once installed (see Installation above), the wildweb console script is what MCP clients launch over stdio.

Point your client at the venv's executable, not a bare command

An MCP client launches the server as a plain subprocess using exactly the command string in its own config — that launch does not go through your shell, so it never activates the virtualenv you installed into and often doesn't even inherit your normal shell PATH. A bare "command": "wildweb" only works if wildweb happens to resolve on whatever PATH that client's process has, which is unreliable. The robust fix, used throughout every example below: give the client the venv's own executable path directly (the ~/.venvs/wildweb venv from Installation), instead of relying on PATH resolution:

{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}

(~/.venvs/wildweb/Scripts/wildweb.exe on Windows — adjust the path in every example below the same way.) No args needed for this alone, no PATH dependency, nothing to activate. Replace /Users/you/.venvs/wildweb with your own venv's actual path.

Config: none, a file, or an environment variable

The simplest entry needs no config at all — the server runs with its built-in defaults:

{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb"
}

To use a config file instead, add --config with an absolute path (the client usually launches the server from its own working directory, not this project's):

{
  "command": "/Users/you/.venvs/wildweb/bin/wildweb",
  "args": ["--config", "/absolute/path/to/config.json"]
}

Alternatively, set WILDWEB_CONFIG as an environment variable on the entry instead of passing --config.

Claude Desktop

Add a wildweb entry under mcpServers in Claude Desktop's config file (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows). No config (built-in defaults):

{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb"
    }
  }
}

With a config file:

{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"]
    }
  }
}

Restart Claude Desktop for the change to take effect.

Claude Code

Add it as a user- or project-scoped server with the claude mcp add CLI. The first wildweb is the server's name within Claude Code; the second is the command it runs. No config:

claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb

With a config file:

claude mcp add wildweb /Users/you/.venvs/wildweb/bin/wildweb -- --config /absolute/path/to/config.json

Or add the equivalent mcpServers block shown above directly to the project's .mcp.json.

Kiro CLI

Add a wildweb entry under mcpServers in Kiro's MCP config file — globally at ~/.kiro/settings/mcp.json, or per-project at .kiro/settings/mcp.json (workspace settings take precedence when both exist). No config:

{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "disabled": false,
      "autoApprove": []
    }
  }
}

With a config file:

{
  "mcpServers": {
    "wildweb": {
      "command": "/Users/you/.venvs/wildweb/bin/wildweb",
      "args": ["--config", "/absolute/path/to/config.json"],
      "disabled": false,
      "autoApprove": []
    }
  }
}

Kiro does not inherit your shell's PATH at all, which is exactly why the command above is the venv's full path rather than a bare wildweb. Run /mcp inside Kiro CLI to confirm the server loaded.

opencode

Add a local server under mcp in opencode.jsonc (project root) or the global ~/.config/opencode/opencode.jsonc. No config:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": ["/Users/you/.venvs/wildweb/bin/wildweb"]
    }
  }
}

With a config file:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "wildweb": {
      "type": "local",
      "command": [
        "/Users/you/.venvs/wildweb/bin/wildweb",
        "--config",
        "/absolute/path/to/config.json"
      ]
    }
  }
}

Or add it from the CLI. No config:

opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb

With a config file:

opencode mcp add wildweb -- /Users/you/.venvs/wildweb/bin/wildweb --config /absolute/path/to/config.json

Other clients

Any MCP client that launches servers over stdio (Cursor, Windsurf, etc.) accepts the same command/args shape under whatever key its own config format uses for MCP servers — consult that client's docs for the exact file location.