ScrapingBee Python SDK

ScrapingBee is a web scraping API that handles headless browsers and rotates proxies for you. The Python SDK makes it easier to interact with ScrapingBee's API.

Installation

You can install ScrapingBee Python SDK with pip.

pip install scrapingbee

Usage

The ScrapingBee Python SDK is a wrapper around the requests library.

Signup to ScrapingBee to get your API key and some free credits to get started.

Table of Contents

• HTML API
  • Auto-Mode
• Google Search API
• Fast Search API
• Amazon API
• Walmart API
• YouTube API
• ChatGPT API
• Gemini API
• Usage API

HTML API

The HTML API allows you to scrape any webpage and get the HTML content.

Basic Request

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')

response = client.html_api(
    'https://www.scrapingbee.com',
    params={
        'render_js': False,
    }
)

print(response.content)

Making a POST request

response = client.html_api(
    'https://httpbin.org/post',
    method='POST',
    data={
        'key': 'value'
    }
)

Auto-Mode

With Auto-Mode, ScrapingBee picks the cheapest scraping configuration that successfully scrapes the page for you: it tries the cheaper options first and stops at the first one that works. You are charged only for the winning configuration (and 0 credits if every configuration fails).

>>> from scrapingbee import ScrapingBeeClient

>>> client = ScrapingBeeClient(api_key='REPLACE-WITH-YOUR-API-KEY')

# Auto-Mode: ScrapingBee picks the cheapest config that works; you're charged only for the winning one.
>>> response = client.html_api(
    'https://example.com',
    method='GET',
    params={
        'mode': 'auto',
        # Optional: cap the credits a single request may cost (omit for uncapped).
        'max_cost': 25
    }
)

# Spb-auto-cost reports the credits actually charged (0 if every config failed).
>>> response.headers['Spb-auto-cost']
'1'

Notes:

• Auto-Mode is only available on GET requests.
• max_cost is optional and must be >= 1; omit it to leave the cost uncapped.
• mode=auto cannot be combined with render_js, premium_proxy, or stealth_proxy (ScrapingBee chooses these for you). Sending them together returns a 400.

Google Search API

Scrape Google search results in real-time.

response = client.google_search(
    search='web scraping tools',
    params={
        'language': 'en',
        'country_code': 'us',
        'nb_results': 10
    }
)

print(response.json())

Fast Search API

Lightweight Google search results in under a second.

response = client.fast_search(
    search='pizza in new york',
    params={
        'country_code': 'us',
        'language': 'en',
        'page': 1
    }
)

print(response.json())

Amazon API

Scrape Amazon search results, product details, and pricing.

Amazon Search

response = client.amazon_search(
    query='laptop',
    params={
        'domain': 'com',
        'language': 'en',
        'pages': 1
    }
)

print(response.json())

Amazon Product

response = client.amazon_product(
    query='B0D2Q9397Y',  # ASIN
    params={
        'domain': 'com'
    }
)

print(response.json())

Amazon Pricing

response = client.amazon_pricing(
    asin='B0DPDRNSXV',
    params={
        'domain': 'com',
        'light_request': True
    }
)

print(response.json())

Walmart API

Scrape Walmart search results and product details.

Walmart Search

response = client.walmart_search(
    query='laptop',
    params={
        'sort_by': 'best_match',
        'device': 'desktop'
    }
)

print(response.json())

Walmart Product

response = client.walmart_product(
    product_id='123456789',
    params={
        'device': 'desktop'
    }
)

print(response.json())

YouTube API

Scrape YouTube search results, video metadata, and subtitles.

YouTube Search

response = client.youtube_search(
    search='web scraping tutorial',
    params={
        'sort_by': 'relevance',
        'type': 'video'
    }
)

print(response.json())

YouTube Metadata

response = client.youtube_metadata(video_id='dQw4w9WgXcQ')
print(response.json())

YouTube Subtitles

response = client.youtube_subtitles(
    video_id='dQw4w9WgXcQ',
    params={
        'language': 'en',
        'subtitle_origin': 'uploader_provided'
    }
)
print(response.json())

ChatGPT API

Use ChatGPT with optional web search.

response = client.chatgpt(
    prompt='What is web scraping?',
    params={
        'search': True,
        'country_code': 'us'
    }
)

print(response.json())

Gemini API

Send prompts to Gemini and receive AI-generated responses.

response = client.gemini(
    prompt='Best programming languages for data science',
    params={
        'country_code': 'us',
        'add_html': False
    }
)

print(response.json())

Usage API

Check your API credit usage.

response = client.usage()
print(response.json())
# {
#     "max_api_credit": 8000000,
#     "used_api_credit": 1000023,
#     "max_concurrency": 200,
#     "current_concurrency": 1
# }

Legacy Methods (Deprecated)

The get() and post() methods are deprecated and will be removed in a future version. Please use html_api() instead.

# Deprecated
client.get(url, params={...})

# Use instead
client.html_api(url, method='GET', params={...})

Screenshot

Here is a little example on how to retrieve and store a screenshot from the ScrapingBee blog.

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')

response = client.html_api(
    'https://www.scrapingbee.com/',
    params={
        'screenshot': True,
        'screenshot_full_page': True,
        'window_width': 375,
    }
)

with open('screenshot.png', 'wb') as f:
    f.write(response.content)

Retries

The client includes a retry mechanism for 5XX responses.

client.html_api(url, params={...}, retries=5)

Using ScrapingBee with Scrapy

Scrapy is the most popular Python web scraping framework. You can easily integrate ScrapingBee's API with the Scrapy middleware.