scrapewise

Python client for the ScrapeWise web-scraping and price-monitoring API. One runtime dependency (httpx), fully type-hinted, synchronous.

Install

pip install scrapewise

Quickstart

from scrapewise import ScrapewiseClient

with ScrapewiseClient(api_key="YOUR_SCRAPEWISE_API_KEY") as sw:   # or env SCRAPEWISE_API_KEY
    groups = sw.list_groups()                                      # GET /scraper/group/list
    rows = sw.get_sample_data(sw.list_scrapers()[0]["id"])          # GET /scraper/{id}/get-sample-data
    print(len(groups), len(rows))

Authentication

Where to get a key: sign in at portal.scrapewise.ai and create one under Settings → API Keys. Two scopes exist:

Your sign-in password is not a key, and a key created anywhere else is rejected with 401. If reads work but creating or running a scraper returns 401, the key is LLM_READ and you need an LLM_FULL one.

Pass api_key= explicitly, or set SCRAPEWISE_API_KEY in the environment and construct with no arguments. The key is sent as Authorization: Bearer <key>. A missing key raises ScrapewiseConfigurationError at construction time instead of failing later with a 401.

import os
os.environ["SCRAPEWISE_API_KEY"] = "YOUR_SCRAPEWISE_API_KEY"
sw = ScrapewiseClient()

Self-hosted or staging deployments: pass base_url= or set SCRAPEWISE_BASE_URL. The default is https://portal.scrapewise.ai/api/scraper-api/api.

Timeouts

The default timeout is 60 seconds, chosen to match the hard tool-call ceiling of the hosted ScrapeWise MCP gateway so that code written against this client behaves identically when driven from an agent. The REST API itself has no such ceiling, so you can raise it:

sw = ScrapewiseClient(timeout=180.0)           # client-wide
sw.get_sample_data(scraper_id, timeout=120.0)  # per call

One route is slow by nature: load_site() renders a live page and routinely exceeds 60 s, so it defaults to its own 180 s timeout (scrapewise.LOAD_SITE_TIMEOUT) rather than the client default.

What is wrapped

Every method maps to exactly one documented REST route. Nothing is inferred.

Two helpers are built on top of those routes and add no new ones: iter_group_data() pages through get_group_data(), and latest_run() returns the newest get_load_history() entry.

Errors

All failures raise a subclass of ScrapewiseError, which carries status_code, the raw body, the request method/url, and a payload property that parses the body as JSON when possible.

from scrapewise import ScrapewiseAuthError, ScrapewiseError

try:
    sw.list_scrapers()
except ScrapewiseAuthError as exc:
    print("bad key:", exc.status_code)
except ScrapewiseError as exc:
    print(exc.status_code, exc.payload)

400 → ScrapewiseBadRequestError, 401/403 → ScrapewiseAuthError, 404 → ScrapewiseNotFoundError, 409 → ScrapewiseConflictError, 429 → ScrapewiseRateLimitError, 5xx → ScrapewiseServerError. Network failures raise ScrapewiseTransportError, timeouts the ScrapewiseTimeoutError subclass.

Platform behaviour worth knowing

These are properties of the ScrapeWise API, not of this client, and are repeated in the relevant docstrings:

• get_sample_data() returns at most 100 rows. It is a sample, not an export — use latest_run()["itemsQuantity"] for the real row count of a run.
• Paginated responses key rows on content, not items.
• replace_site_links() replaces the whole link list for a site. Links not present in the call are removed.
• attach_schema() is effectively one-way: it switches the scraper's extractor to the AI/schema path.
• Writing group post-process rules is optimistically locked. Read returns version; the write expects expectedVersion. update_group_post_process_rules() does that rename for you.
• A scraper is scoped to one domain. Posting links from another host fails with LINK_HOST_MISMATCH.

Development

python -m venv .venv && . .venv/bin/activate
pip install -e ".[test]"
pytest

Tests mock HTTP at the transport layer with respx; there are no live network calls in the suite.

License

MIT © BEBOTECH OÜ