scrapy-mcp-server 0.1.0

MCP server for Scrapy request inspection and debugging

Scrapy MCP Server

MCP (Model Context Protocol) server for Scrapy request inspection and debugging.

Installation & Usage

Use uvx to run the MCP server:

uvx scrapy-mcp-server

Configuration

Add to your MCP client configuration (e.g., Claude Desktop):

For Claude Desktop on MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json

For Claude Desktop on Windows: %APPDATA%/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "scrapy-inspector": {
      "command": "uvx",
      "args": ["scrapy-mcp-server"]
    }
  }
}

Prerequisites

Install the required Scrapy middleware in your Scrapy project:

pip install scrapy-mcp-middleware

Add to your Scrapy settings.py:

SPIDER_MIDDLEWARES = {
    'scrapy_mcp_middleware.SessionTrackerMiddleware': 100,
}

SESSION_STORE_URL = "http://127.0.0.1:8765"

Features

• Request Tree: View all captured requests in hierarchical structure
• Request Details: Inspect headers, bodies, and response data
• Selector Testing: Test CSS/XPath selectors interactively
• Spider Development: Get comprehensive spider development guides

Requirements

• Python >= 3.8
• Compatible with Windows, Linux, and macOS (Intel & Apple Silicon)

License

Copyright © 2015–2025 CoreDump Engineering. All rights reserved.

Support

For issues and questions, visit the GitHub repository.