scraping-mcp 0.2.0

MCP server for Scraping — REST API wrapper for LLM agents.

scraping-mcp

MCP (Model Context Protocol) server for Scraping — lets LLM agents (Cursor, Claude Desktop, Claude Code, Zed) call our scraping tools directly.

Install

pip install scraping-mcp
# or with pipx for isolation:
pipx install scraping-mcp

Requires Python 3.11+.

Hosted alternative (no install)

If you don't want to install Python locally, point your MCP client at the hosted instance:

{
  "mcpServers": {
    "scraping": {
      "url": "https://api.scraping.io/mcp/",
      "transport": "http",
      "headers": {
        "Authorization": "Bearer fc_live_..."
      }
    }
  }
}

The hosted version exposes the same 7 tools as the stdio package.

Setup

Get an API key at dashboard/keys.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) / %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": {
        "SCRAPING_API_KEY": "fc_live_abc123...",
        "SCRAPING_BASE_URL": "https://api.scraping.io"
      }
    }
  }
}

Restart Claude Desktop. Tools will appear as "scraping".

Cursor

Create .cursor/mcp.json in the repo root:

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}

Claude Code

Add to ~/.claude.json (user scope):

{
  "mcpServers": {
    "scraping": {
      "command": "scraping-mcp",
      "env": { "SCRAPING_API_KEY": "fc_live_abc123..." }
    }
  }
}

Tools

Tool	Description
scrape(url, format?, cookies?, timeout_s?)	Scrape single URL. Returns markdown or {status:pending, task_id} on timeout.
crawl(urls[], cookies?, timeout_s?)	Batch-scrape up to 500 URLs. Returns items or {status:pending, crawl_id} on timeout.
cancel_crawl(crawl_id)	Cancel pending URLs of a crawl batch + refund credits.
get_scrape_status(task_id)	Check status of a scrape task.
get_crawl_status(crawl_id, include_items?)	Check status + items of a crawl batch.
map(url, max_urls?, include_subdomains?)	List a site's URLs (sitemap.xml + robots.txt + fallback crawl). 1 credit.
get_balance()	Current credit balance.

Examples

Agent: "Scrape the README at https://github.com/hashicorp/terraform"
-> calls scrape(url="https://github.com/hashicorp/terraform")
-> returns {status: "finished", result: {markdown: "# Terraform..."}}

Agent: "Scrape these 50 docs pages and summarize"
-> calls crawl(urls=[...50 urls...])
-> returns {status: "finished", total: 50, items: [...]}

Agent: "Check my credit balance"
-> calls get_balance()
-> returns {balance: 87}

Environment variables

Name	Required	Default	Description
SCRAPING_API_KEY	yes	—	Your API key (starts with fc_live_)
SCRAPING_BASE_URL		http://localhost:8000	Scraping API endpoint
SCRAPING_SCRAPE_TIMEOUT_S		60	Scrape tool wait-timeout
SCRAPING_CRAWL_TIMEOUT_S		600	Crawl tool wait-timeout
SCRAPING_REQUEST_TIMEOUT_S		30	Individual HTTP request timeout

Troubleshooting

Server not starting: Check SCRAPING_API_KEY is set in MCP client config env. Logs go to stderr — Claude Desktop / Cursor show them in their MCP logs.

"connection refused": Verify SCRAPING_BASE_URL points to a running Scraping API. For local dev: docker compose up in the monorepo.

Agent receives {status: "pending", task_id}: Timeout exceeded. Agent should call get_scrape_status(task_id) (or get_crawl_status) to retrieve the result once ready. Scrape typically takes 2-20 seconds; heavy sites (JS-rendered via L3) can take 30-60 seconds.

License

MIT.

Links

• Homepage
• REST API docs
• Report issues