ScrapeUnblocker Cloud - Python client & CLI

Deploy, run, schedule and pull data from Scrapy spiders hosted on ScrapeUnblocker Cloud, from Python or the command line.

ScrapeUnblocker Cloud is managed Scrapy hosting: you push an unmodified Scrapy project, buy concurrency slots and run it. The difference from other Scrapy clouds is that traffic goes through ScrapeUnblocker by default, so a spider that gets blocked elsewhere runs here without you writing any proxy or fingerprint code (--unblock).

pip install scrapeunblocker-cloud

No runtime dependencies: this package is pure standard library, so installing it next to your Scrapy project never pulls in a conflicting dependency tree.

Get a token

Create an API token in the ScrapeUnblocker portal under Spider Cloud. Then either run su-cloud login, or set the environment:

export SU_CLOUD_TOKEN="sk_..."
export SU_CLOUD_ORG="acme"
export SU_CLOUD_PROJECT="shop"     # defaults to the current directory name

CLI

su-cloud login                              # store url/token/org/project in ~/.su-cloud.json
su-cloud deploy                             # package the current Scrapy project and build it
su-cloud run products -a category=shoes --unblock --wait
su-cloud jobs --state running
su-cloud logs j1699999999abc --lines 200
su-cloud logs j1699999999abc --level WARNING      # why did it fail? searches the whole log
su-cloud logs j1699999999abc --contains "403"
su-cloud log-download j1699999999abc        # the whole log as j1699999999abc.log.gz
su-cloud items j1699999999abc > out.jsonl   # page scraped items to stdout as JSON lines
su-cloud download j1699999999abc -o out.jsonl.gz   # download the whole dataset
su-cloud stats j1699999999abc              # crawl statistics
su-cloud schedules
su-cloud schedule-add nightly "0 2 * * *" products --unblock
su-cloud destinations
su-cloud destination-add mymongo mongodb -s uri=mongodb+srv://... -s database=shop -s collection=products

Library

from scrapeunblocker_cloud import SpiderCloudClient

client = SpiderCloudClient(token="sk_...", org="acme", project="shop")

# deploy the current directory, then run a spider through ScrapeUnblocker
client.deploy(".", notes="add price field")
job = client.run("products", args={"category": "shoes"}, unblock=True)

job.wait()                       # block until it finishes
print(job.state, job.items_total)

for item in job.items():         # stream the scraped data straight back
    print(item)

for line in job.logs(lines=100):
    print(line)

Org and project set on the client are the defaults for every call, so single-project users never repeat themselves. Every method also accepts org= / project= to override per call.

What you can reach

Errors

Everything raises a subclass of SpiderCloudError: AuthError (bad token), NotFoundError (unknown org/project/job), APIError (other non-2xx, with .status and .detail), ConnectionFailed (control plane unreachable) and ConfigError (missing token/org/project).

License

MIT