scrapely-mcp-python

Scrapely MCP server for Python. It exposes Scrapely Actors, storage, runs, tasks, and documentation to MCP clients over stdio and streamable HTTP transports, with dynamic per-Actor tools, MCP-Apps widget support, Skyfire/x402 agentic payments, Segment telemetry, and Sentry instrumentation.

Install

python -m venv .venv && source .venv/bin/activate
pip install -e .
# The local Scrapely API client is expected as a sibling checkout:
pip install -e ../scrapely-client-python

Run (stdio)

export SCRAPELY_TOKEN=...   # or keep a token in ~/.scrapely/credentials.json
scrapely-mcp-server --actors=username/actor-name
scrapely-mcp-server --tools=docs                      # public tools only, no token needed
scrapely-mcp-server --tools=actors,runs,storage       # category selection
scrapely-mcp-server --ui=apps                         # force MCP Apps (widget) mode

Dev server (stdio, or streamable HTTP on localhost with --http):

python -m scrapely_mcp.dev_server --http --port 8000

Library surface

from scrapely_mcp import ActorsMcpServer, create_stateless_server

• ActorsMcpServer(options) — the facade: load_tools_from_input(input, client), add_tool, remove_tool, connect(transport), close().
• create_stateless_server(input, options) — per-request composition for stateless hosts.

Tool categories

actors, docs, runs, storage, tasks, dev — selected via --tools/Input.tools; Actor full names (e.g. scrapely/web-scraper) are also valid selectors and become dynamic tools generated from the Actor's real input schema.

Implementation notes

• MCP SDK: official Python mcp package (low-level Server).
• Validation: jsonschema with AJV-compatible semantics (array coercion, additional-property stripping, ReDoS keyword removal) — see utils/ajv.py.
• Web widgets: the React widget bundle stays JavaScript; the built HTML assets are shipped under scrapely_mcp/web/ and served as MCP resources (resources/widgets.py).
• Payments: Skyfire and x402 agentic-payment providers; payment error payloads (e.g. x402 402 payment-required responses) are surfaced directly by the client (utils/payment_errors.py).
• Eval harness (evals/): intentionally not included in this package; the unit test suites live under tests/ (pytest).
• Logging: a stdlib wrapper provides the structured logging call surface, including softFail and the log-aggregator "error"-word sanitization.

Tests

pip install -e .[dev]
pytest