scrapy-mcp

A headless web-scraping MCP server built on Scrapy. It exposes Scrapy's scraping primitives — polite fetching, CSS/XPath extraction, link and table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls — as MCP tools an agent can call over stdio.

• Headless, no rendering. Pages are fetched and parsed as HTML; no browser, no JavaScript execution. This keeps the footprint tiny — it runs comfortably on weak machines.
• Reactor-safe. Every operation runs in a short-lived Scrapy subprocess, so Twisted's reactor never lives inside the asyncio MCP server (no ReactorNotRestartable), and memory is reclaimed after each call.
• Polite by default. Obeys robots.txt, throttles with AutoThrottle, and enforces hard page/depth caps so a crawl can't run away.

Install / run

Run straight from PyPI with uv — no install step:

uvx scrapy-mcp

Or install it:

uv pip install scrapy-mcp
scrapy-mcp

The server speaks MCP over stdio. Point any MCP client at it. For Claude Desktop, add to claude_desktop_config.json:

{
  "mcpServers": {
    "scrapy": {
      "command": "uvx",
      "args": ["scrapy-mcp"]
    }
  }
}

Tools

Selector format (extract / start_crawl)

selectors maps an output field to a selector. Each value is either a CSS string (first match) or an object for more control:

{
  "title": "h1::text",
  "price": "span.price::text",
  "all_links": {"css": "a::attr(href)", "all": true},
  "first_heading": {"xpath": "//h1/text()"}
}

"all": true returns every match as a list; otherwise the first match is returned.

Crawls are asynchronous

start_crawl returns immediately with a job_id. The crawl runs as a detached worker that streams results to disk, so it survives a server restart. Poll crawl_status(job_id), then read items with crawl_results(job_id) (safe to call mid-crawl for partial results). Jobs are stored under the system temp dir and reclaimed after 7 days (configurable).

Configuration

All settings are optional environment variables (sensible, polite defaults tuned for a weak host). They're how you tune a uvx scrapy-mcp deployment.

Development

uv venv
uv pip install -e ".[dev]"
uv run pytest          # unit tests (no network)
uv build               # build wheel + sdist into dist/

License

MIT © Eitan Hadar