scrapy-mcp-middleware 0.1.0

Scrapy middleware for MCP request tracking

Scrapy MCP Middleware

Scrapy middleware for tracking requests and responses with MCP (Model Context Protocol) integration.

Installation

pip install scrapy-mcp-middleware

Usage

Add the middleware to your Scrapy settings:

SPIDER_MIDDLEWARES = {
    'scrapy_mcp_middleware.SessionTrackerMiddleware': 100,
}

# Optional: Configure the session store URL (defaults to http://127.0.0.1:8765)
SESSION_STORE_URL = 'http://127.0.0.1:8765'

Features

• Tracks all requests and responses
• Hierarchical request tree tracking
• Base64 encoding for request/response bodies
• Configurable session store URL
• Automatic storage cleanup on spider start

Requirements

• Python >= 3.8
• Scrapy >= 2.0.0
• requests >= 2.25.0