web-scraping-api-sdk

A Python client for the ScrapingBee web scraping API. One class covers the HTML API, Auto-Mode cost control, and nine dedicated scraper endpoints that return parsed JSON instead of HTML you have to select against.

Independent client, published separately from ScrapingBee's own scrapingbee SDK. You need a ScrapingBee API key: app.scrapingbee.com, 1,000 free credits on signup, no card.

Every endpoint, parameter, credit cost and response field documented below was verified against the live API on 2026-08-25.

Install

pip install web-scraping-api-sdk

Requires Python 3.8+ and requests.

Quickstart

from web_scraping_api_sdk import ScrapingBeeAPI

api = ScrapingBeeAPI("YOUR-API-KEY")

page = api.scrape("https://news.ycombinator.com", render_js=False)
print(page.status_code, page.cost, page.request_id)
print(page.text[:500])

Authentication

The client sends Authorization: Bearer YOUR-API-KEY on every request. ScrapingBee still accepts an api_key query parameter, but its documentation marks that method deprecated, so this client does not use it.

Cost control with Auto-Mode

Picking a proxy tier by hand means overpaying on pages that need nothing special, or getting blocked on pages that need a lot. auto() asks ScrapingBee to try configurations from cheapest to most expensive and charge only for the one that actually returned the page.

page = api.auto("https://example.com", max_cost=25)
print(page.auto_cost)   # 1, 5, 10 or 25, or 0 if every configuration failed

max_cost caps the climb: 25 reaches premium proxy with JavaScript and never touches the 75-credit stealth tier. If no configuration succeeds, the request costs 0 credits.

The HTML API

scrape() fetches any public URL and accepts the full parameter set from the ScrapingBee documentation.

JavaScript rendering. On by default. A real headless browser executes the page, which is what makes single-page apps and lazy-loaded content work. Turn it off with render_js=False to drop from 5 credits to 1. Control timing with wait, wait_for (a CSS selector) or wait_browser.

Proxies. Rotating datacenter IPs by default. premium_proxy=True switches to residential IPs, which is also what country_code geotargeting requires. stealth_proxy=True uses the anti-bot tier. session_id keeps the same IP across a sequence of requests.

Output shape. Ask for text with return_page_text=True, markdown with return_page_markdown=True (useful for RAG and LLM pipelines), a PNG with screenshot(), or structured JSON via data extraction rules:

page = api.scrape(
    "https://news.ycombinator.com",
    extract_rules={"titles": {"selector": ".titleline > a", "type": "list"}},
)
print(page.json())

AI extraction. Describe the fields in plain language instead of writing selectors, via AI web scraping. Adds 5 credits.

page = api.scrape("https://example.com", ai_query="the main heading and the first paragraph")

Dedicated scrapers

Nine endpoints return parsed JSON, so there are no selectors to maintain when a layout changes. Response fields listed below are the real top-level keys returned by the live API.

Google search API

results = api.google("web scraping api", country_code="us").json()
for r in results["organic_results"]:
    print(r["position"], r["title"], r["url"])

Returns organic_results, ai_overviews, top_ads, bottom_ads, shopping_ads, related_queries, questions, top_stories, news_results, local_results, hotel_results, knowledge_graph, related_searches and meta_data. Costs 15 credits, or 10 with light_request=True.

One search_type parameter switches surface:

search_type	Returns	Scraper page
classic	Organic results, the default	Google scraper
news	News results	Google News API
maps	Local listings	Google API docs
images	Image results	Google Images scraper
lens	Visual matches for an image URL	Google Lens API
shopping	Product listings, sortable and price-filtered	Google Shopping API
ai_mode	The AI-generated answer	Google AI Mode API
ads	Paid placements	Google Ads API

news is unavailable with device="mobile", lens expects an image URL as the search term, and ai_mode accepts at most 400 characters. Shopping supports sort_by, min_price and max_price, priced in the currency implied by country_code.

Fast Search API

api.fast_search("web scraping api").json()

A lighter search endpoint returning answer_box, organic, people_also_ask, related_searches, top_stories and search_parameters. Flat 10 credits. See Fast Search.

Amazon scraper API

search = api.amazon_search("laptop stand").json()
asin = search["products"][0]["asin"]

product = api.amazon_product(asin).json()
print(product["title"], product["buybox"], product["rating"])

pricing = api.amazon_pricing(asin).json()

amazon_search() returns products, products_count, refinements, page and url. amazon_product() takes an ASIN or a product URL as query and returns asin, brand, bullet_points, buybox, category, coupon, currency, delivery, description and more. An unknown ASIN returns HTTP 404 with {"error": "Product not found"}. All three cost 5 credits, or 15 with light_request=False.

Related pages: Amazon search, ASIN API, review API, offers API, best sellers, seller API, Amazon feature page.

Walmart scraper API

api.walmart_search("coffee maker").json()
api.walmart_product("10450114").json()

walmart_product() returns price, price_strikethrough, currency, rating, rating_count, fulfillment, out_of_stock, gtin, seller_id, images and more. Search supports start_page, sort_by, min_price, max_price, store_id and delivery_zip. Costs 10 credits, or 15 with light_request=False.

Related pages: Walmart price API, search results, inventory, seller API, Walmart feature page.

YouTube scraper API

api.youtube_search("web scraping tutorial").json()
meta = api.youtube_metadata("dQw4w9WgXcQ").json()
subs = api.youtube_subtitles("dQw4w9WgXcQ").json()

Metadata and subtitles take a video id, not a URL. Metadata returns title, description, channel_id, channel_url, duration, like_count, comment_count, upload_date, categories, tags, thumbnails, formats, age_limit and is_live. Subtitles returns subtitles. Search filters include type, duration, upload_date, sort_by, hd, 4k, live, creative_commons and subtitles. Flat 5 credits.

Related pages: YouTube video scraper, transcript scraper, comment scraper, captions API, YouTube feature page.

ChatGPT and Gemini

api.chatgpt("what is a web scraping api").json()
api.gemini("what is a web scraping api").json()

ChatGPT returns results_text, results_markdown, results_json, full_html, llm_model and prompt. Gemini returns results_text, results_markdown, citations, full_html and prompt. Both cost 15 credits. ScrapingBee notes ChatGPT citations are not returned on every call, so treat them as best effort. See ChatGPT and Gemini.

Shopee and agentic search

api.shopee("https://shopee.co.id/product-name-i.93014939.1881883105")
api.agentic_search("senior data scientists at large tech companies in the US", limit=10)

Shopee costs 75 credits per product page and takes a product URL. Agentic employee search costs 3,750 credits per call, so check usage() before putting it in a loop. Both parameter sets come from the official documentation. The Shopee endpoint returned HTTP 500 for the placeholder URL used in the docs, charged at 0 credits, which is the documented behaviour for 500 responses; agentic search was not run because of its cost.

Credit reference

Configuration	Credits
Rotating proxy, no JavaScript	1
Rotating proxy with JavaScript (default)	5
Premium proxy, no JavaScript	10
Premium proxy with JavaScript	25
Stealth proxy with JavaScript	75
ai_query or ai_extract_rules	+5
Google, Walmart	10 light, 15 standard
Amazon	5 light, 15 standard
YouTube	5
Fast Search	10
ChatGPT, Gemini	15
Shopee	75
Agentic employee search	3,750

Check consumption with api.usage(), which returns max_api_credit, used_api_credit, max_concurrency, current_concurrency and renewal_subscription_date. It is free and limited to 6 calls per minute. Full pricing at scrapingbee.com/pricing.

Response metadata

page.cost           # Spb-cost, credits charged
page.auto_cost      # Spb-auto-cost, only on auto() calls, 0 when all tiers failed
page.request_id     # Spb-request-id, quote this in support requests
page.resolved_url   # Spb-resolved-url, final URL after redirects
page.status_code
page.headers

Errors

Non-2xx responses raise ScrapingBeeError carrying status_code, body and request_id. HTTP 500 responses are not charged credits, so retrying on 500 is safe.

from web_scraping_api_sdk import ScrapingBeeAPI, ScrapingBeeError

try:
    page = api.scrape("https://example.com", stealth_proxy=True)
except ScrapingBeeError as error:
    print(error.status_code, error.request_id)

Scope

Public, pre-login pages only. Scraping behind login credentials is prohibited by the ScrapingBee terms. Keep your API key out of shared environments, including AI coding assistants.

Links

• ScrapingBee documentation
• Pricing and credits
• Examples in eight languages

MIT licensed.