ScrapeBadger

ScrapeBadger Python SDK

PyPI version Python versions License Tests Coverage Code style: ruff Type checked: mypy

The official Python SDK for ScrapeBadger - async web scraping APIs for Twitter, Google, Vinted, Reddit, and more.

Features

  • Async-first - Built with asyncio for high-performance concurrent scraping
  • Type-safe - Full type hints and Pydantic models for all responses
  • Automatic pagination - Iterator methods with smart rate limit handling
  • Resilient retries - Exponential backoff on transient errors
  • 37+ Twitter endpoints - Tweets, users, lists, communities, trends, geo, real-time streams
  • 19 Google product APIs - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, Local Pack, Shorts, Flights, Products
  • Vinted scraping - Search items, item details, user profiles, brands, colors, markets
  • Reddit scraping - Search posts/subreddits/users/domains, subreddit posts, post comments, user profiles, trophies, wiki pages, moderators
  • Web scraping - Anti-bot bypass, JS rendering, and AI data extraction

Installation

pip install scrapebadger

Or with uv:

uv add scrapebadger

Quick Start

import asyncio
from scrapebadger import ScrapeBadger

async def main():
    async with ScrapeBadger(api_key="your-api-key") as client:
        # Get a user profile
        user = await client.twitter.users.get_by_username("elonmusk")
        print(f"{user.name} has {user.followers_count:,} followers")

        # Scrape a website
        result = await client.web.scrape("https://scrapebadger.com", format="markdown")
        print(result.content)

        # Search tweets
        tweets = await client.twitter.tweets.search("python programming")
        for tweet in tweets.data:
            print(f"@{tweet.username}: {tweet.text[:100]}...")

asyncio.run(main())

Authentication

Get your API key from scrapebadger.com and pass it to the client:

from scrapebadger import ScrapeBadger

client = ScrapeBadger(api_key="sb_live_xxxxxxxxxxxxx")

You can also set the SCRAPEBADGER_API_KEY environment variable:

export SCRAPEBADGER_API_KEY="sb_live_xxxxxxxxxxxxx"

Available APIs

API Description Documentation
Web Scraping Scrape any website with JS rendering, anti-bot bypass, and AI extraction Web Scraping Guide
Twitter 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams Twitter Guide
Google 19 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products Google Guide
Vinted Search items, item details, user profiles, brands, colors, statuses, and markets Vinted Guide
Reddit Search posts, subreddits, users, and domains; fetch post comments, subreddit rules, moderators, wiki pages, user trophies Reddit Guide
Instagram User profile/about/related/posts/videos/reels/tagged/pinned/followers/following/stories/highlights, media detail/comments/replies/likers/oEmbed, search (users/hashtags/places/top/reels/music/autocomplete), hashtag/location/audio feeds Instagram Guide
Amazon 14 endpoints — search, autocomplete, product detail, offers, reviews, bestsellers, new releases, deals, category browse, seller profile/products/feedback, markets, categories Amazon Guide
eBay 13 endpoints across 18 markets — search, search by image (visual search), completed/sold search, item detail, item reviews, seller profile/items/feedback, category browse, categories, autocomplete, markets eBay Guide
YouTube 39 endpoints — search, autocomplete, video detail/related/comments/replies/transcript/captions/streams/live-chat/batch, channel detail + videos/shorts/streams/playlists/community/about/subscriber-count/in-channel-search/resolve, playlists/items/mixes, trending/hashtag/home, shorts, community post/comments, music search, oembed, categories/languages/regions YouTube Guide
TikTok 25 endpoints — user profile/videos/followers/following/liked/reposts, video detail/comments/replies/related/transcript/oEmbed, search (general/videos/hashtags/users), music detail/videos, hashtag detail/videos, trending videos/hashtags/songs, ad library, regions TikTok Guide
Immobiliare 8 endpoints across immobiliare.it, indomio.es, indomio.gr, immotop.lu — autocomplete, search, listing detail, agency profile/listings, price stats, markets, reference Immobiliare Guide
LoopNet 5 endpoints across loopnet.com/.ca/.co.uk/.fr/.es — commercial-real-estate search (for-lease/for-sale/auctions), listing detail, broker profile, markets, property types LoopNet Guide
Apartments.com US rental listings with UNIT-LEVEL pricing — search by location with bed/price filters, plus per-unit rent, beds, baths, sqft and availability for every rentable unit Apartments Guide
Walmart 11 endpoints (US-only) — search, category browse, deals feed, autocomplete, product detail, reviews, seller profile/products, store detail, markets Walmart Guide
Baidu 4 endpoints — web search (language + date filters), news vertical, image search, autocomplete. Results carry the real target URL, not just Baidu's tracking redirect Baidu Docs
Bing 6 endpoints — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets Bing Docs
DuckDuckGo 7 endpoints — web search (with abstract box), image/news/video search, autocomplete, instant answers, regions DuckDuckGo Docs
Yahoo 6 endpoints across 35 markets — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets Yahoo Docs
Yandex 4 endpoints across 6 markets (tr/com/ru/by/kz/uz) — web search (organic + ads + sitelinks + inline media), image search, reverse-image (CBIR) search, markets Yandex Docs
ChatGPT Prompt the real chatgpt.com anonymously — structured answer with citations anchored to character offsets, the full retrieved search trail, and AEO/GEO brand-visibility analysis ChatGPT Guide
Gemini Prompt the real gemini.google.com anonymously — structured answer with cited web sources, the full retrieved search trail, and AEO/GEO brand-visibility analysis Gemini Docs

Error Handling

from scrapebadger import (
    ScrapeBadger,
    ScrapeBadgerError,
    AuthenticationError,
    RateLimitError,
    InsufficientCreditsError,
    NotFoundError,
    ValidationError,
    ServerError,
)

async with ScrapeBadger(api_key="your-key") as client:
    try:
        user = await client.twitter.users.get_by_username("elonmusk")
    except AuthenticationError:
        print("Invalid API key")
    except RateLimitError as e:
        print(f"Rate limited. Retry after {e.retry_after} seconds")
        print(f"Limit: {e.limit}, Remaining: {e.remaining}")
    except InsufficientCreditsError:
        print("Out of credits! Purchase more at scrapebadger.com")
    except NotFoundError:
        print("User not found")
    except ValidationError as e:
        print(f"Invalid parameters: {e}")
    except ServerError:
        print("Server error, try again later")
    except ScrapeBadgerError as e:
        print(f"API error: {e}")

Configuration

Custom Timeout and Retries

from scrapebadger import ScrapeBadger

client = ScrapeBadger(
    api_key="your-key",
    timeout=120.0,      # Request timeout in seconds (default: 300)
    max_retries=5,      # Retry attempts (default: 10)
)

Advanced Configuration

from scrapebadger import ScrapeBadger
from scrapebadger._internal import ClientConfig

config = ClientConfig(
    api_key="your-key",
    base_url="https://scrapebadger.com",
    timeout=300.0,
    connect_timeout=10.0,
    max_retries=10,
    retry_on_status=(500, 502, 503, 504),
    headers={"X-Custom-Header": "value"},
)

client = ScrapeBadger(config=config)

Retry Behavior

The SDK automatically retries requests that fail with 500, 502, 503, or 504 status codes, as well as transport-level failures (timeouts, network errors, dropped connections), using exponential backoff (1s, 2s, 4s, 8s, ...). Each retry logs a warning:

⚠ 503 Service Unavailable — retrying in 4s (attempt 3/10)

To see these warnings, configure Python logging:

import logging
logging.basicConfig(level=logging.WARNING)

Rate Limit Aware Pagination

When using *_all pagination methods, the SDK reads X-RateLimit-Remaining and X-RateLimit-Reset headers from each response. When remaining requests drop below 20% of your tier's limit, pagination automatically slows down to spread requests across the remaining window — preventing 429 errors. A warning is logged when throttling activates:

⚠ Rate limit: 25/300 remaining (resets in 42s), throttling pagination to ~0.6 req/s

This works transparently with all tier levels (Free: 60/min, Basic: 300/min, Pro: 1000/min, Enterprise: 5000/min).

Development

Setup

# Clone the repository
git clone https://github.com/scrape-badger/scrapebadger-python.git
cd scrapebadger-python

# Install dependencies with uv
uv sync --dev

# Install pre-commit hooks
uv run pre-commit install

Running Tests

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=src/scrapebadger --cov-report=html

# Run specific tests
uv run pytest tests/test_client.py -v

Code Quality

# Lint
uv run ruff check src/ tests/

# Format
uv run ruff format src/ tests/

# Type check
uv run mypy src/

# All checks
uv run ruff check src/ tests/ && uv run ruff format --check src/ tests/ && uv run mypy src/

Contributing

Contributions are welcome! Please read our Contributing Guide for details.

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Make your changes
  4. Run tests and linting (uv run pytest && uv run ruff check)
  5. Commit your changes (git commit -m 'Add amazing feature')
  6. Push to the branch (git push origin feature/amazing-feature)
  7. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

Made with ❤️ by ScrapeBadger

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for dipsydegen from gravatar.com dipsydegen

Credits

Author/Maintainer: ScrapeBadger

License

MIT License (MIT)

Requires

Python >=3.10

Provides Extra

dev

Tags

api async data-extraction scraping sdk social-media twitter web-scraping

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapebadger-0.43.1.tar.gz (209.9 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

scrapebadger-0.43.1-py3-none-any.whl (339.0 kB view details)

Uploaded Python 3

File details

Details for the file scrapebadger-0.43.1.tar.gz.

File metadata

  • Download URL: scrapebadger-0.43.1.tar.gz
  • Upload date:
  • Size: 209.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for scrapebadger-0.43.1.tar.gz
Algorithm Hash digest
SHA256 7be88160501cc5724833746a143494558ee9445b6996e9a7a4e5e299876c4128
MD5 7d37353c768d4b42ba71e4dd07467ae5
BLAKE2b-256 8a9327775f358bdab8007a390e2afb0ea72ca3dd356714544e0ef81276a9236d

See more details on using hashes here.

Provenance

The following attestation bundles were made for scrapebadger-0.43.1.tar.gz:

Publisher: publish.yml on scrape-badger/scrapebadger-python

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file scrapebadger-0.43.1-py3-none-any.whl.

File metadata

  • Download URL: scrapebadger-0.43.1-py3-none-any.whl
  • Upload date:
  • Size: 339.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for scrapebadger-0.43.1-py3-none-any.whl
Algorithm Hash digest
SHA256 93453ce15c70cded137bc604dbb02bcac3731551058b4d853ac9d02b548ddb8b
MD5 e7deee961f01717ea51d5e086e083396
BLAKE2b-256 91ac298974107598c7b4756921db4d004cb5a7c07a087267ca5c01053548527d

See more details on using hashes here.

Provenance

The following attestation bundles were made for scrapebadger-0.43.1-py3-none-any.whl:

Publisher: publish.yml on scrape-badger/scrapebadger-python

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.43.1 This release

2 files

0.43.0

2 files

0.42.0

2 files

0.41.0

2 files

0.40.1

2 files

0.40.0

2 files

0.39.0

2 files

0.38.1

2 files

0.38.0

2 files

0.37.1

2 files

0.37.0

2 files

0.36.0

2 files

0.35.0

2 files

0.34.0

2 files

0.33.1

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.0

2 files

0.29.0

2 files

0.28.0

2 files

0.27.1

2 files

0.27.0

2 files

0.26.1

2 files

0.26.0

2 files

0.25.0

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.0

2 files

0.22.0

2 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.0

2 files

0.15.7

2 files

0.15.6

2 files

0.15.5

2 files

0.15.3

2 files

0.15.2

2 files

0.15.1

2 files

0.15.0

2 files

0.14.0

2 files

0.13.1

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.3

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.0

2 files

0.3.1

2 files

0.3.0

2 files

0.2.0

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page