scrape-n-search MCP server

Unified web-search and scraping tools for AI agents, backed by multiple providers (TinyFish, Tavily, Exa, ZenRows, Scrappey, Firecrawl, Serper, Jina). The agent calls a handful of tools and never needs to know which provider served the request.

Because every provider fetches on its own servers, the client's IP (e.g. a VPN IP that target sites reject) is never seen by the target site.

Architecture

MCP client (agent)
   │  MCP stdio protocol
   ▼
MCPServer (server.py) ── 7 tools: web_search, web_fetch, web_browser_actions,
                                 web_screenshot, web_session, web_crawl, web_extract
   │
   ▼
Router (router.py) ── picks provider by priority + capability, fails over on error
   │
   ├── TinyFish   (search, fetch)
   ├── Tavily     (search, fetch)
   ├── Exa        (search, fetch)
   ├── ZenRows    (fetch, extract)
   ├── Scrappey   (fetch, browser_actions, screenshot, session)
   ├── Firecrawl  (search, fetch, crawl, extract)
   ├── Serper     (search)
   └── Jina       (search*, fetch)

• Normalization: every adapter maps its provider-specific response into the shared models (SearchResult, FetchedPage, CrawlPage), so results look the same regardless of provider.
• Selection: the router uses the first enabled provider (API key present) that supports the requested tool, in priority order.
• Failover: if the chosen provider errors, the router tries the next candidate and aggregates errors if all fail. The response always names the provider that served it.
• Default provider: DEFAULT_PROVIDER forces a preferred provider for every call (the agent can still override per-call with the provider parameter). When set, tools that provider cannot serve are hidden from the client.

Tools

Every tool returns a JSON string. All accept an optional provider parameter to force a specific provider; otherwise the default/auto-selected provider is used.

web_search

Search the web and return ranked results.

Response: {query, provider, results: [{title, url, snippet, published_date?, score?}]}

web_fetch

Fetch a URL to clean content, server-side.

Response: {provider, url, title?, format, content}

web_browser_actions

Run a sequence of browser actions on a URL (Scrappey full-browser mode) and return the page content plus any execute_js results. Served by Scrappey only.

Response: {provider, url, format, content, javascript_return[]}

See the scrappey://playbook resource (or src/scrape_n_search/playbook.md) for recipes — e.g. reading a full AliExpress item description (scroll + click "View more" + extract).

web_screenshot

Capture a screenshot of a URL (Scrappey browser mode), returned as base64 PNG. Served by Scrappey only.

Response: {provider, url, format: "png", screenshot_base64}

web_session

Manage Scrappey browser-session lifecycle (Scrappey-only). Create a session, attach requests to it via the session parameter of web_fetch / web_browser_actions / web_screenshot, and destroy it when done.

Response: {provider, op, session, solution}

Known limitation (verified 2026-10): the create/destroy lifecycle works, but Scrappey currently destroys a named session once a request.get completes, so cookies/browser state do not persist across calls yet despite the vendor docs. Treat session as a lifecycle-managed name; re-create sessions as needed.

web_crawl

Recursively crawl a domain. Served by Firecrawl only.

Response: {provider, pages: [{url, title?, content?}]}

web_extract

Extract structured JSON from a page. Served by Firecrawl (prompt-based) and ZenRows (extract=auto).

Response: {provider, url, data}

Provider coverage

browser_actions and screenshot are Scrappey-only capabilities (full-browser features). * s.jina.ai (search) now requires an API key; r.jina.ai (fetch) still works without one.

Configuration

API keys and settings live in mcp-server/.env (auto-loaded at startup; gitignored). Copy .env.example to .env and fill in your keys. The server loads .env from the working directory and falls back to the project's own mcp-server/.env, so it works regardless of how it is launched. Existing environment variables take precedence over .env.

Example — Scrappey as the only/default provider (fetch + browser actions + screenshots):

# mcp-server/.env
SCRAPPEY_API_KEY=your_key_here
DEFAULT_PROVIDER=scrappey   # hides web_search/web_crawl/web_extract

Scrappey

Scrappey is the cheapest scraping option in the research (€0.10/1K direct, €1/1K full browser; pay-as-you-go, no subscription, balance never expires, residential proxies included, pay-per-success). It provides fetch, browser actions, screenshots, and session management (web_session) in this MCP.

Full documentation of its two modes, pricing, API, browser actions, and behavior: docs/scrappey.md. A common-actions playbook for agents is exposed as the scrappey://playbook MCP resource (source: src/scrape_n_search/playbook.md).

Setup

python3 -m venv .venv
.venv/bin/pip install -e mcp-server

Run

.venv/bin/scrape-n-search            # stdio MCP server
# or: .venv/bin/python -m scrape_n_search

MCP client config

Claude Desktop / Claude Code claude_desktop_config.json or .mcp.json:

{
  "mcpServers": {
    "scrape-n-search": {
      "command": "/abs/path/to/scrape-n-search/.venv/bin/python",
      "args": ["-m", "scrape_n_search"],
      "cwd": "/abs/path/to/scrape-n-search/mcp-server",
      "env": {
        "SCRAPPEY_API_KEY": "..."
      }
    }
  }
}

Oh My Pi (OMP) integration

Registered on this machine as a user-level MCP server in ~/.omp/agent/mcp.json (scrape-n-search, stdio). After editing that file (or .env), reload in the OMP session:

/mcp reload
/mcp list        # confirm scrape-n-search is discovered
/mcp test scrape-n-search

The server auto-loads mcp-server/.env regardless of the working directory, so keys added there apply without touching the OMP config.

Testing

.venv/bin/python -m pytest mcp-server/tests -q

Unit tests cover per-provider normalization (mocked HTTP), router selection/failover, and the MCP surface end-to-end over an in-memory transport.

Development — adding a provider

1. Create src/scrape_n_search/providers/<name>.py with a subclass of BaseProvider implementing the tools it supports (see base.py and an existing adapter). Normalize responses into the shared models.
2. Register it in src/scrape_n_search/providers/__init__.py (build_providers + DEFAULT_PRIORITY).
3. Add the env var to .env.example and the coverage table above.
4. Add a normalization test to tests/test_providers.py (mocked HTTP via httpx.MockTransport).