ScrapeMCP — Web Scraping MCP Server

Servidor MCP para extracción estructurada de datos web. Scrapea páginas, tablas, listas, sitemaps y más. Incluye protección SSRF integrada.

Features / Funcionalidades

SSRF Protection / Protección SSRF

El servidor bloquea automáticamente accesos a:

• IPs privadas (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
• Localhost/loopback (127.0.0.0/8, ::1)
• Link-local (169.254.0.0/16, fe80::/10)
• Hostnames bloqueados: localhost, metadata.google.internal, 169.254.169.254
• Dominios .internal y .local

Solo permite esquemas http:// y https://.

Tech Stack

• Python — >=3.11
• Framework: mcp (FastMCP) via stdio JSON-RPC
• HTTP: httpx (async) — replaced sync requests for non-blocking I/O
• Parsing: beautifulsoup4 + html5lib
• Export: CSV (sanitized), Markdown, JSON
• Config: python-dotenv — reads .env file for all settings

Quick Start

# Instalar dependencias
pip install mcp requests beautifulsoup4 html5lib httpx

# Ejecutar servidor
python server.py

Ejemplos

# Scrapear página completa
result = await session.call_tool("scrape", {"url": "https://example.com"})

# Scrapear con selectores personalizados
result = await session.call_tool("scrape", {
    "url": "https://example.com",
    "selectors": {"title": "h1", "price": ".price"}
})

# Inspeccionar estructura de página
result = await session.call_tool("inspect", {"url": "https://example.com"})

# Extraer tablas
result = await session.call_tool("tables", {
    "url": "https://example.com",
    "selector": "table"
})

# Scrapear lista
result = await session.call_tool("scrape_list", {
    "url": "https://example.com/items",
    "item_selector": ".item",
    "fields": {"name": "h2", "price": ".price"}
})

# Scrapear recursivamente
result = await session.call_tool("scrape_recursive", {
    "start_url": "https://example.com/blog",
    "link_selector": "a.post-link",
    "item_selector": "article",
    "fields": {"title": "h1", "content": "p"},
    "max_pages": 10
})

# Exportar a CSV
result = await session.call_tool("export", {
    "data": '[{"name": "Alice", "age": 30}]',
    "format": "csv"
})

Project Structure

scrapemcp/
├── server.py              # MCP server entry point (tools)
├── scrapers/
│   ├── __init__.py
│   ├── base.py            # BaseScraper, ScrapeResult, SSRF validation
│   ├── page.py            # PageScraper (scrape, inspect)
│   ├── table.py           # TableScraper (tables)
│   ├── list_scraper.py    # ListScraper (scrape_list, scrape_recursive)
│   └── sitemap.py         # SitemapScraper (sitemap, scrape_sitemap)
├── exporters.py           # CSV, Markdown, JSON export
├── client.py              # Test client CLI
└── pyproject.toml

🔧 Recent Improvements

• SSRF Bypass Fixed — DNS resolution added: URL-encoded private IPs are now properly blocked
• Async HTTP — Replaced sync requests with httpx.AsyncClient (non-blocking, async-compatible)
• State Race Fixed — Removed shared mutable _last_url/_last_soup (no more cross-call contamination)
• .env Support — Reads HTTP_TIMEOUT, SITEMAP_URL_LIMIT, MAX_RECURSIVE_PAGES from environment
• Sitemap Discovery — Tries robots.txt for Sitemap: before falling back to /sitemap.xml
• Recursive Timeout — 120s wall-clock timeout prevents runaway crawling
• Configurable Rate Limit — Recursive crawl delay via RECURSIVE_DELAY env var
• Export Size Limit — Rejects data > 10MB before parsing