scraping-mcp

MCP (Model Context Protocol) server for Scraping — lets LLM agents (Cursor, Claude Desktop, Claude Code, Zed) call our scraping tools directly.

Install

pip install scraping-mcp
# or with pipx for isolation:
pipx install scraping-mcp

Requires Python 3.11+.

Hosted alternative (no install)

If you don't want to install Python locally, point your MCP client at the hosted instance:

{
  "mcpServers": {
    "scraping": {
      "url": "https://api.scraping.io/mcp/",
      "transport": "http",
      "headers": {
        "Authorization": "Bearer fc_live_..."
      }
    }
  }
}

The hosted version exposes the same 7 tools as the stdio package.

Setup

Get an API key at dashboard/keys.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) / %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": {
        "SCRAPING_API_KEY": "fc_live_abc123...",
        "SCRAPING_BASE_URL": "https://api.scraping.io"
      }
    }
  }
}

Restart Claude Desktop. Tools will appear as "scraping".

Cursor

Create .cursor/mcp.json in the repo root:

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}

Claude Code

Add to ~/.claude.json (user scope):

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}

Tools

Examples

Agent: "Scrape the README at https://github.com/hashicorp/terraform"
-> calls scrape(url="https://github.com/hashicorp/terraform")
-> returns {status: "finished", result: {markdown: "# Terraform..."}}

Agent: "Scrape these 50 docs pages and summarize"
-> calls crawl(urls=[...50 urls...])
-> returns {status: "finished", total: 50, items: [...]}

Agent: "Check my credit balance"
-> calls get_balance()
-> returns {balance: 87}

Environment variables

Troubleshooting

Server not starting: Check SCRAPING_API_KEY is set in MCP client config env. Logs go to stderr — Claude Desktop / Cursor show them in their MCP logs.

"connection refused": Verify SCRAPING_BASE_URL points to a running Scraping API. For local dev: docker compose up in the monorepo.

Agent receives {status: "pending", task_id}: Timeout exceeded. Agent should call get_scrape_status(task_id) (or get_crawl_status) to retrieve the result once ready. Scrape typically takes 2-20 seconds; heavy sites (JS-rendered via L3) can take 30-60 seconds.

License

MIT.

Links

• Homepage
• REST API docs
• Report issues