scrapingpros

Production web scraping at scale, without the plumbing. Submit a list of URLs, stream results as the workers finish them, and let the server handle the hard parts — browsers, proxies, retries, anti-bot, soft-block detection. Submit 50,000 URLs in one call, walk away, come back to handled results.

📚 Full docs · 📦 PyPI · 📝 Release notes · 🔧 API status

Install

pip install scrapingpros

Requires Python 3.10+. No signup needed — the demo token below gives you 5,000 credits/month, 30 req/min, all features enabled.

The headline use case — submit_batch() + iter_results()

Streaming async batches are the centerpiece of the SDK. They scale from a handful of URLs up to tens of thousands, with progress, automatic resume, soft-block detection, and per-job traceability via custom_id. Benchmarked at N=1000 with browser=True, this is 5× faster than the now-removed scrape_many (185 s vs 930 s) and more reliable (998 / 1000 vs 990 / 1000).

import asyncio
from scrapingpros import AsyncClient

async def main():
    async with AsyncClient("demo_6x595maoA6GdOdVb") as client:
        # Submit 50,000 URLs in one call — the API queues them across the worker pool.
        batch = await client.submit_batch("daily-products", [
            {"url": product_url, "custom_id": product_id}
            for product_id, product_url in catalog.items()
        ])

        # Stream results as workers finish them. Progress, ETA, and counters are live.
        async for result in batch.iter_results():
            if result.guidance.success:
                save(product_id=result.custom_id, content=result.content)
            else:
                log_failure(result.custom_id, result.guidance.error_type, result.guidance.next_steps)

            # Live progress — no extra API calls
            print(f"{batch.pct:.1%}  ({batch.success_count}/{batch.total})  ETA {batch.eta_seconds}s")

asyncio.run(main())

What you get out of the box:

• Streaming, not "submit and poll". Results come back the moment each worker finishes — your loop body runs in real time.
• custom_id round-trips through every layer (request → job → result → webhook). Map results back to your domain objects without depending on order.
• result.guidance.success is the server's verdict on whether the page produced usable content. It catches soft-blocks (Google CAPTCHA pages with 200 + large body, Amazon "Robot Check") that a naive status_code check would miss.
• Automatic refunds. Credits are refunded on the spot for any response that doesn't deliver usable content (4xx/5xx, captchas, worker failures, timeouts). You pay for successful content only.
• Per-job retries with proxy rotation and IP/fingerprint changes. retry_on_block=True handles anti-bot sites without you writing retry logic.
• Worker-restart resilient. The SDK re-attaches transparently to in-flight runs across transient ConnectionError and worker churn.

Common patterns

Downloading files (PDFs, images, binaries) — v0.6.0+

resp = client.scrape("https://investors.example.com/charter.pdf", browser=False)
if resp.is_binary:
    resp.save("charter.pdf")              # one-liner: writes to disk
    # or: data = resp.body                 # bytes, mirrors requests.Response.content
else:
    print(resp.content)                   # text response (markdown / html)

resp.content_type carries the MIME (e.g. "application/pdf", "image/png"). Works inside submit_batch / batch_scrape too — every yielded ScrapeResponse has the same accessors.

Browser-rendered + anti-bot protected sites

result = client.scrape(
    "https://spa-site.com",
    browser=True,            # 5 credits, full JS rendering
    retry_on_block=True,     # auto-retry up to 3x with different IP/fingerprint
)

The API picks the right engine internally per target — you don't configure browser engines.

Structured data extraction (CSS / XPath, no parsing on your side)

result = client.scrape("https://quotes.toscrape.com/", extract={
    "quotes":  {"selector": "css:.text",   "multiple": True},
    "authors": {"selector": "css:.author", "multiple": True},
})
print(result.extracted_data["quotes"])

Form-encoded POST (OAuth2, legacy APIs)

from scrapingpros import MethodPOST

resp = client.scrape(token_url, http_method=MethodPOST(
    payload={"grant_type": "client_credentials"},
    content_type="form",   # since v0.5.0; default is "json"
))

Wait for hidden DOM nodes (<script> tags with embedded JSON)

from scrapingpros import WaitForSelectorAction

result = client.scrape(url, browser=True, actions=[
    WaitForSelectorAction(selector="css:script#__NEXT_DATA__", time=8000, state="attached"),
])

Capture response bodies (auth tokens, GraphQL payloads)

from scrapingpros import NetworkCaptureConfig

result = client.scrape(url, browser=True, network_capture=NetworkCaptureConfig(
    resource_types=["xhr", "fetch"],
    url_pattern="*identitytoolkit.googleapis.com*",   # captures matching response bodies (≤64 KB)
))
for entry in result.network_requests or []:
    if "body" in entry:
        token = parse_token(entry["body"])

List return instead of streaming — batch_scrape()

Same server-side scaling as submit_batch(), simpler signature. Drop-in replacement for scrape_many() (removed in v0.7.0):

results = client.batch_scrape([
    {"url": u, "custom_id": product_id, "browser": True}
    for product_id, u in catalog.items()
])
for r in results:
    if r.guidance.success:
        save(r.custom_id, r.content)

Crash-resilient pipelines

For long-running scrapers that can't afford to lose track of a batch:

from scrapingpros import SyncClient, SubmitTimeout

client = SyncClient("...")

# Since v0.5.3, submit_batch automatically generates an Idempotency-Key
# UUID per call, so retrying a SubmitTimeout returns the SAME collection
# (server dedupes within 24h — no duplicate run, no double cost).
try:
    batch = client.submit_batch(name, items, submit_timeout=30.0)
except SubmitTimeout:
    batch = client.submit_batch(name, items)   # safe — server replays

For belt-and-braces, use find_recent_batch to verify before retrying. It uses server-side ?name=&since= filters and reattaches to the live run automatically:

import uuid
from datetime import datetime, timezone

batch_name = f"daily-{uuid.uuid4().hex[:8]}"
fired_at = datetime.now(timezone.utc)

try:
    batch = client.submit_batch(batch_name, items)
except SubmitTimeout:
    batch = client.find_recent_batch(name=batch_name, since=fired_at)
    if batch is None:
        batch = client.submit_batch(batch_name, items)

# Reattaching from a persisted (cid, rid) on a separate process:
for r in client.iter_results(saved_cid, saved_rid):
    save(r.custom_id, r.content)

If a get_job_result raises one of the typed 404 subclasses (JobResultPending, JobResultExpired, JobResultLost, JobNotFound), each tells you exactly what to do next — no string parsing required. See Batch API docs.

Pricing — pay only for usable content

1 simple request = 1 credit, 1 browser request = 5 credits. Credits are refunded automatically on failures (is_success=false). Anti-bot, proxy rotation, and per-country routing are included on every plan.

Check your usage at any time:

client.scrape(url)
print(client.quota_remaining, "credits left")
print(client.billing())

Error handling

from scrapingpros import SyncClient, AuthenticationError, RateLimitError, QuotaExceededError

try:
    result = client.scrape(url)
except AuthenticationError:
    print("Invalid token — use demo_6x595maoA6GdOdVb for testing")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except QuotaExceededError:
    print("Monthly quota exceeded — upgrade your plan for more requests")

All SDK exceptions inherit from ScrapingProsError.

Configuration

client = AsyncClient(
    "demo_6x595maoA6GdOdVb",            # or your dedicated token / SP_TOKEN env var
    base_url="https://api.scrapingpros.com",   # default
    timeout=120.0,                       # request timeout in seconds
    max_retries=3,                       # auto-retry on 429
)

A blocking SyncClient exists with the same surface for REPL sessions, notebooks, and one-off scripts that need a single result. Inside a running event loop it emits a RuntimeWarning — use AsyncClient there.

More

• Full documentation with end-to-end recipes for batches, JS execution, viability tests, and anti-bot strategies.
• Release notes for every version.
• API reference for AI agents — single-file llms-full.txt for Claude / GPT / Cursor.
• MCP server for AI agents: Model Context Protocol integration with 6 tools and anti-injection protection.

License

MIT