scrapely-mcp

An MCP server for Scrape.ly. It gives AI agents and MCP clients access to a Scrape.ly account: search the Store for Actors, read their README and input schema, run them, follow the runs, and read the results from datasets, key-value stores and logs. It can also manage saved tasks and search the Scrape.ly and Crawlee documentation.

The same package runs in two ways:

• Hosted: scrapely-mcp-http, a multi-tenant streamable HTTP server. Scrape.ly runs it behind the API at https://api.scrape.ly/mcp. Most users connect to that URL and never install anything.
• Local: scrapely-mcp-server, a stdio server started by a desktop MCP client on your machine.

Connect to the hosted server

Point your client at https://api.scrape.ly/mcp.

• Clients that support MCP authorization find the OAuth flow on their own and ask you to sign in.
• Otherwise send your API token as a header: Authorization: Bearer <token>.
• Clients that cannot set headers can append ?token=<token> to the URL. URLs tend to end up in logs, so prefer the header.

Example for a client that takes a JSON config:

{
  "mcpServers": {
    "scrapely": {
      "url": "https://api.scrape.ly/mcp",
      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
    }
  }
}

Run locally (stdio)

Requires Python 3.11 or newer.

uvx --from scrapely-mcp scrapely-mcp-server
# or
pip install scrapely-mcp
scrapely-mcp-server

The token comes from SCRAPELY_TOKEN, or from the token field of ~/.scrapely/credentials.json or ~/.scrapely/auth.json. A token is required unless every selected tool works without one (search-actors, fetch-actor-details and the docs tools).

{
  "mcpServers": {
    "scrapely": {
      "command": "uvx",
      "args": ["--from", "scrapely-mcp", "scrapely-mcp-server"],
      "env": { "SCRAPELY_TOKEN": "YOUR_TOKEN" }
    }
  }
}

Options:

Tools

Default categories: actors, docs, runs, storage, tasks, automations. The dev category (report-problem) is off by default.

Whenever call-actor, get-actor-run or an Actor tool is enabled, get-actor-run, get-actor-output, get-dataset-items, get-key-value-store-record and abort-actor-run are added as well.

call-actor waits up to waitSecs (0 to 45, default 30) and returns the run status and storage IDs; the agent then fetches the results with get-actor-output (pass the runId, or a datasetId). It returns a page of dataset items (offset, limit up to 1000, default 100, and fields with dot notation) plus the run's OUTPUT record. Long text values are cut and a page may hold fewer items than limit so the reply stays small; hasMore and nextOffset say where to continue. get-dataset-items remains available for finer control. Clients that support protocol tasks can run call-actor as a long-running task and collect the result when the run ends. Progress notifications are sent while a run is awaited, and cancelling the request aborts the run.

Any Scrape.ly API GET endpoint can also be read with resources/read using its full URL, for example https://api.scrape.ly/v2/datasets/<id>/items?limit=20.

Hosted server

scrapely-mcp-http --host 0.0.0.0 --port 8000

URL parameters on the MCP endpoint: tools, actors, maxActorMemoryBytes, ui, payment (skyfire or x402), telemetry-enabled, token.

Each session gets its own server instance bound to the SHA-256 fingerprint of the credential that opened it; a request with a different credential is answered as if the session did not exist. SCRAPELY_TOKEN is never used by the HTTP server.

A container image definition is in Dockerfile (docker build --build-arg SOURCE=local -t scrapely-mcp .).

Configuration

Telemetry is off unless it is enabled and a write key is configured; the package ships no keys. Logs never contain tokens, payment tokens or payment signatures.

Widgets

MCP Apps widget tools (search-actors-widget, fetch-actor-details-widget, call-actor-widget, get-actor-run-widget) are implemented but disabled by default, and no widget HTML is bundled. With widgets disabled every client gets the plain tool set.

Development

uv venv && uv pip install -e ".[test]"
pytest

License

MIT, see LICENSE. Third-party notices are in NOTICE.