scrapyio-sdk

Official Python SDK for Scrapy.io.

Server-side only. Keep API keys off client devices.

Install

pip install scrapyio-sdk

Authentication

import os
from scrapyio_sdk import ScrapyIO

client = ScrapyIO(api_key=os.environ["SCRAPYIO_API_KEY"])

Always sends Authorization: Bearer <api_key>.

First request

tools = client.tools.list(q="instagram", limit=20)
# {"items", "total", "offset", "limit"}

tool = client.tools.get("datadoping", "instagram-profile-scraper")
# or:
same = client.tool("datadoping/instagram-profile-scraper").get()

Run a scraper

Sync (/v1/api)

result = client.tool("datadoping/instagram-profile-scraper").call({
    "username": "nasa",
})

Async (/v1/scraper) + poll

run = client.tool("datadoping/instagram-profile-scraper").start({
    "usernames": ["nasa"],
})

finished = client.run(run["id"]).wait(
    poll_interval_ms=3000,
    timeout_ms=15 * 60_000,
)

page = client.run(run["id"]).list_items(offset=0, limit=100)

Schedules

schedule = client.schedules.create(
    {
        "publisher": "datadoping",
        "slug": "instagram-profile-scraper",
        "runName": "nightly-ig",
        "timezone": "UTC",
        "frequency": "one-time",
        "date": "2026-09-20",
        "time": "23:50",
        "inputs": ["nasa"],
    },
    idempotency_key="nightly-ig-v1",
)

client.schedules.update(schedule["id"], {"isActive": False})
client.schedules.delete(schedule["id"])

Errors

from scrapyio_sdk import ScrapyIO, ScrapyIOError

try:
    client.tools.get("nope", "missing")
except ScrapyIOError as err:
    # err.type, err.status, err.message, err.doc_url
    ...

Mental model

client
├── tools.list / tools.get
├── tool("publisher/slug").get / .call / .start
├── runs.list / runs.get
├── run(id).get / .wait / .list_items
└── schedules.list / get / create / update / delete

Development

cd sdk/python
python -m pip install -e ".[dev]"
pytest

Notes

• GET requests may retry on transient 5xx/429; execution POSTs are not auto-retried.
• list_items() is JSON-only in v0.1.
• Companion JS package: @scrapyio/sdk.