ScrapeRack SDK

Python client for submitting trusted Python functions to a self-hosted ScrapeRack cluster.

ScrapeRack is alpha software. Use it on trusted infrastructure and expect breaking changes.

Install

pip install scraperack

Run a function

The environment must already exist on the cluster.

from scraperack import Cluster


def add(a, b):
    from scraperack_runtime import secret, variable

    print(variable("LOG_LEVEL", "INFO"))
    database_url = secret("DATABASE_URL")
    return a + b


cluster = Cluster("http://controller:7070")
try:
    execution = cluster.run(
        "python",
        add,
        (2, 3),
        variables={"LOG_LEVEL": "DEBUG"},
        secrets={"DATABASE_URL": "postgresql://..."},
    )
    print(cluster.result(execution["id"]))
finally:
    cluster.close()

Installation, cluster setup, and source code are available at github.com/damadjan/scraperack.