scrapingdog 1.0.0

Python SDK for the Scrapingdog web scraping and SERP APIs.

Scrapingdog Python Library & Package

Integrate web scraping and search data into your AI workflow, RAG / fine-tuning, or Python application using this official-style wrapper for Scrapingdog.

Scrapingdog supports general web scraping (rotating & residential proxies, JavaScript rendering), Google, Google Maps, Google Shopping, Bing, Baidu, DuckDuckGo, Amazon, Walmart, eBay, App Stores, LinkedIn, and more.

Query a vast range of data at scale, including web pages, search results, product listings, flight and hotel data, job postings, social profiles, and more.

Installation

To install the scrapingdog package, simply run the following command:

$ pip install scrapingdog

Requires Python 3.8+ and depends only on requests.

Simple Usage

Let's start by searching for Coffee on Google:

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google(query="coffee")

print(results)

The results variable now contains a ScrapingdogResults object, which acts just like a standard dictionary, with attribute-style access added on top (results.organic_results).

Scraping an arbitrary web page returns the raw HTML instead:

html = client.scrape("https://example.com", dynamic=True, premium=False)

The Scrapingdog API key can be obtained from scrapingdog.com/signup.

Environment variables are a secure, safe, and easy way to manage secrets. Set export SCRAPINGDOG_API_KEY=<secret_api_key> in your shell. The client reads this variable automatically when api_key is omitted.

Error handling

Unsuccessful requests raise scrapingdog.HTTPError or scrapingdog.TimeoutError exceptions. The returned status code reflects the sort of error that occurred; please refer to the Scrapingdog documentation for more details.

import os
import scrapingdog

# A default timeout can be set here.
client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"), timeout=10)

try:
    results = client.google(query="coffee")
except scrapingdog.HTTPError as e:
    if e.status_code == 401:   # Invalid API key
        print(e.message)
    elif e.status_code == 400: # Missing required parameter
        pass
    elif e.status_code == 429: # Exceeds the plan's throughput limit
        pass
except scrapingdog.TimeoutError as e:
    # Handle timeout
    print(f"The request timed out: {e}")

All exceptions inherit from scrapingdog.ScrapingdogError.

Documentation

Full API documentation is available on scrapingdog.com.

Every Scrapingdog scraper has a dedicated method (75 in total). Each is a thin wrapper over client.get(), so any documented query parameter can also be passed as a keyword argument — booleans are auto-converted to true/false and None values are dropped. For any endpoint without a named method, call get() directly with the path:

data = client.get("google_trends", query="bitcoin", geo="US")

Basic Examples in Python

General Web Scraping

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
html = client.scrape(
    "https://example.com",
    dynamic=True,     # render JavaScript
    premium=True,     # use premium/residential proxies
    country="us",
)

• API Documentation: scrapingdog.com/web-scraping-api

Search Google

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google(query="coffee", country="us", results=10, page=0)

• API Documentation: scrapingdog.com/google-search-api

Search Bing

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.bing(query="coffee", cc="us")

• API Documentation: scrapingdog.com/bing-search-api

Search Baidu

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.baidu(query="coffee")

• API Documentation: scrapingdog.com/baidu-search-api

Search DuckDuckGo

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.duckduckgo(query="coffee")

• API Documentation: scrapingdog.com/duckduckgo-search-api

Search Google Maps

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_maps(query="pizza")

• API Documentation: scrapingdog.com/google-maps-api

Search Google Shopping

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_shopping(query="shoes", country="us")

• API Documentation: scrapingdog.com/google-shopping-api

Search Google News

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_news(query="football", country="us")

• API Documentation: scrapingdog.com/google-news-api

Search Google Scholar

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_scholar(query="coffee")

• API Documentation: scrapingdog.com/google-scholar-api

Search Google Jobs

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_jobs(query="jobs in london")

• API Documentation: scrapingdog.com/google-jobs-api

Google Trends

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.google_trends(query="pizza,burger", data_type="TIMESERIES")

• API Documentation: scrapingdog.com/google-trends-api

Amazon Search

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.amazon_search(query="spoon", domain="com", country="us", page=1)

• API Documentation: scrapingdog.com/amazon-search-scraper

Amazon Product

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.amazon_product(asin="B00AP877FS", domain="com", country="us")

• API Documentation: scrapingdog.com/amazon-product-scraper

Walmart Search

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.walmart_search(url="https://www.walmart.com/search?q=coffee")

• API Documentation: scrapingdog.com/walmart-search-scraper

Search eBay

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.ebay_search(url="https://www.ebay.com/sch/i.html?_nkw=coffee")

• API Documentation: scrapingdog.com/ebay-search-api

Apple App Store

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.apple_app_store(term="whatsapp", country="us", lang="en-us")

• API Documentation: scrapingdog.com/apple-app-store-api

LinkedIn Profile

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.linkedin("rbranson", type="profile")

• API Documentation: scrapingdog.com/profile-scraper-api

YouTube

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.youtube(channel_id="UCX6OQ3DkcsbYNE6H8uQQuVA", country="us")

• API Documentation: scrapingdog.com/youtube-channel-api

Screenshot

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
results = client.screenshot(url="https://www.scrapingdog.com")

• API Documentation: scrapingdog.com/screenshot-api

Account

import os
import scrapingdog

client = scrapingdog.Client(api_key=os.getenv("SCRAPINGDOG_API_KEY"))
account = client.account()  # remaining credits & plan usage

• API Documentation: scrapingdog.com/account-api

Available Methods

All 75 endpoint methods (click to expand)

General & account: scrape, screenshot, account, webhook

Google Search: google, google_images, google_videos, google_shorts, google_news, google_news_v2, google_shopping, google_local, google_jobs, google_autocomplete, google_finance, google_flights, google_hotels, google_lens, google_product, google_immersive_product, google_ai_mode, google_ai_overview, google_ads_transparency

Google Maps: google_maps, google_maps_places, google_maps_photos, google_maps_posts, google_maps_reviews

Google Scholar: google_scholar, google_scholar_profiles, google_scholar_author, google_scholar_cite

Google Patents / Trends: google_patents, google_patents_details, google_trends, google_trends_autocomplete, google_trends_trending_now

Other search engines: bing, bing_shopping, baidu, duckduckgo, yelp, universal_search

Amazon: amazon_search, amazon_product, amazon_offers, amazon_reviews, amazon_autocomplete

Apple: apple_app_store, apple_product, apple_reviews

Retail: walmart_search, walmart_product, walmart_reviews, walmart_autocomplete, ebay_search, ebay_product, flipkart_search, flipkart_product, myntra_search, myntra_product, zillow

Social & professional: linkedin, linkedin_post, linkedin_jobs, indeed, x_profile, x_post, tiktok_profile, tiktok_post, tiktok_ads, facebook, instagram, youtube

AI: chatgpt

License

MIT License.

Contributing

Bug reports and pull requests are welcome. Once dependencies are installed (pip install -e ".[test]"), you can run the tests with pytest.

Publishing a new release

Releases are published to PyPI automatically via GitHub Actions using Trusted Publishing (no API token required).

1. Bump the version in pyproject.toml and scrapingdog/__init__.py.
2. Push a version tag — the release workflow tests, builds, and publishes to PyPI:

   git tag v1.0.0
   git push origin v1.0.0
   

One-time PyPI setup (before the first release): on pypi.org/manage/account/publishing, add a pending publisher with — Project: scrapingdog, Owner: Darshan972, Repository: Scrapingdog-PIP, Workflow: release.yml, Environment: pypi.