web-scraping-api-sdk

A Python client for the ScrapingBee web scraping API. It wraps the HTML API, Auto-Mode cost control, and all nine dedicated scraper endpoints behind one class.

This package talks to ScrapingBee (app.scrapingbee.com). It is an independent client, published separately from ScrapingBee's own scrapingbee SDK. You need a ScrapingBee API key to use it: app.scrapingbee.com, 1,000 free credits on signup.

Install

pip install web-scraping-api-sdk

Quickstart

from web_scraping_api_sdk import ScrapingBeeAPI

api = ScrapingBeeAPI("YOUR-API-KEY")

page = api.scrape("https://news.ycombinator.com", render_js=False)
print(page.status_code, page.cost, page.request_id)
print(page.text[:500])

Authentication uses the Authorization: Bearer header. ScrapingBee still accepts the api_key query parameter but marks it deprecated, so this client does not use it.

Auto-Mode

Choosing a proxy tier by hand means overpaying on easy pages or getting blocked on hard ones. auto() asks ScrapingBee to try configurations from cheapest to most expensive and charge only for the one that works:

page = api.auto("https://example.com", max_cost=25)
print(page.auto_cost)   # one of 1, 5, 10, 25 or 0 if every tier failed

max_cost=25 allows premium proxy with JavaScript but never the 75-credit stealth tier. When every configuration fails, the request costs nothing.

Endpoint reference

Google surfaces

One parameter switches between eight result types:

api.google("web scraping api", search_type="news", country_code="us")
api.google("running shoes", search_type="shopping", min_price=50, sort_by="price_low_to_high")
api.google("how does virtual interlining work", search_type="ai_mode")

Accepted values: classic, news, maps, images, lens, shopping, ai_mode, ads. news is unavailable with device="mobile", lens expects an image URL, and ai_mode caps the query at 400 characters.

Reading the cost

Every response exposes the ScrapingBee headers as attributes:

page = api.scrape("https://example.com")
page.cost          # Spb-cost, credits charged
page.auto_cost     # Spb-auto-cost, present only for auto() calls
page.request_id    # Spb-request-id, quote this in support requests
page.resolved_url  # Spb-resolved-url, the final URL after redirects

Errors

Non-2xx responses raise ScrapingBeeError, carrying the status, the body and the request id. HTTP 500 responses are not charged credits, so retrying on 500 is safe.

from web_scraping_api_sdk import ScrapingBeeAPI, ScrapingBeeError

try:
    page = api.scrape("https://example.com", stealth_proxy=True)
except ScrapingBeeError as error:
    print(error.status_code, error.request_id)

Scope

Public, pre-login pages only. Scraping behind login credentials is prohibited by the ScrapingBee terms. Keep your API key out of shared environments, including AI coding assistants.

Links

• ScrapingBee documentation
• Plans and credits
• Examples in eight languages

MIT licensed.