mcp-server-webcrawl 0.9.0

MCP server for search and retrieval of web crawler content

mcp-server-webcrawl

Bridge the gap between your web crawl and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.

mcp-server-webcrawl provides the LLM a complete menu with which to search your web content, and works with a variety of web crawlers:

• WARC
• wget
• InterroBot
• Katana
• SiteOne

mcp-server-webcrawl is free and open source, and requires Claude Desktop, Python (>=3.10). It is installed on the command line, via pip install:

pip install mcp-server-webcrawl

Features

• Claude Desktop ready
• Fulltext search support
• Filter by type, status, and more
• Multi-crawler compatible
• Supports advanced/boolean and field searching

MCP Configuration

From the Claude Desktop menu, navigate to File > Settings > Developer. Click Edit Config to locate the configuration file, open in the editor of your choice and modify the example to reflect your datasrc path.

You can set up more mcp-server-webcrawl connections under mcpServers as needed.

{
  "mcpServers": {
    "webcrawl": {
      "command": [varies by OS/env, see below],
       "args": [varies by crawler, see below]
    }
  }
}

For step-by-step setup, refer to the Setup Guides.

Windows vs. macOS

On Windows with Python installed on path, the command should simply be mcp-server-webcrawl.

On macOS, you must use the absolute path to the mcp-server-webcrawl executable in the command field, rather than just the command name.

For example:

"command": "/Users/yourusername/.local/bin/mcp-server-webcrawl",

To find the absolute path of the mcp-server-webcrawl executable on your system:

1. Open Terminal
2. Run which mcp-server-webcrawl
3. Copy the full path returned and use it in your config file

wget (using --mirror)

The datasrc argument should be set to the parent directory of the mirrors.

"args": ["--crawler", "wget", "--datasrc", "/path/to/wget/archives/"]

WARC

The datasrc argument should be set to the parent directory of the WARC files.

"args": ["--crawler", "warc", "--datasrc", "/path/to/warc/archives/"]

InterroBot

The datasrc argument should be set to the direct path to the database.

"args": ["--crawler", "interrobot", "--datasrc", "/path/to/Documents/InterroBot/interrobot.v2.db"]

Katana

The datasrc argument should be set to the parent directory of the text cache files.

"args": ["--crawler", "katana", "--datasrc", "/path/to/katana/archives/"]

SiteOne (using archiving)

The datasrc argument should be set to the parent directory of the archives, archiving must be enabled.

"args": ["--crawler", "siteone", "--datasrc", "/path/to/SiteOne/archives/"]

Boolean Search

The query engine supports field-specific (field: value) searches and complex boolean expressions. Fulltext is supported as a combination of the url, content, and headers fields.

While the API interface is designed to be consumed by the LLM directly, it can be helpful to familiarize yourself with the search syntax. Searches generated by the LLM are inspectable, but generally collapsed in the UI. If you need to see the query, expand the MCP collapsable.

Example Queries

Query	Description
privacy	single fulltext search
"privacy policy"	fulltext match exact phrase
privacy*	matches wildcard fulltext results starting with "privacy"
id: 12345	matches a specific resource by ID
url: example.com/*	matches results with URL containing example.com
type: html	HTML pages only
status: 200	matches specific HTTP status code (equal)
status: >=400	matches specific HTTP status code (greater than or equal to)
content: javascript	find javascript in HTTP body (often, but not always HTML)
headers: application/json	match HTTP response headers
privacy AND policy	match both as fulltext search
privacy OR policy	match either as fulltext search
policy NOT privacy	match fullext policies not containing privacy
(login OR signin) AND form	match fullext login or signin with form
type: html AND status: 200	match only HTML pages with HTTP success

Field Search Definitions

Field	Description
id	database ID
url	resource URL
type	enumerated list of types (see types table)
status	HTTP response code of result
headers	HTTP response headers
content	HTTP body—HTML, CSS, JS, and more

Content Types

Type	Description
html	web page
iframe	embedded iframe
img	web image formats
audio	audio files
video	video files
font	web font files
style	CSS stylesheets
script	JavaScript files
rss	RSS syndication feed
text	plain text content
pdf	PDF files
doc	MS Word documents
other	uncategorized