ScrapeBadger Python SDK
The official Python SDK for ScrapeBadger - async web scraping APIs for Twitter, Google, Vinted, Reddit, and more.
Features
- Async-first - Built with
asynciofor high-performance concurrent scraping - Type-safe - Full type hints and Pydantic models for all responses
- Automatic pagination - Iterator methods with smart rate limit handling
- Resilient retries - Exponential backoff on transient errors
- 37+ Twitter endpoints - Tweets, users, lists, communities, trends, geo, real-time streams
- 19 Google product APIs - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, Local Pack, Shorts, Flights, Products
- Vinted scraping - Search items, item details, user profiles, brands, colors, markets
- Reddit scraping - Search posts/subreddits/users/domains, subreddit posts, post comments, user profiles, trophies, wiki pages, moderators
- Web scraping - Anti-bot bypass, JS rendering, and AI data extraction
Installation
pip install scrapebadger
Or with uv:
uv add scrapebadger
Quick Start
import asyncio
from scrapebadger import ScrapeBadger
async def main():
async with ScrapeBadger(api_key="your-api-key") as client:
# Get a user profile
user = await client.twitter.users.get_by_username("elonmusk")
print(f"{user.name} has {user.followers_count:,} followers")
# Scrape a website
result = await client.web.scrape("https://scrapebadger.com", format="markdown")
print(result.content)
# Search tweets
tweets = await client.twitter.tweets.search("python programming")
for tweet in tweets.data:
print(f"@{tweet.username}: {tweet.text[:100]}...")
asyncio.run(main())
Authentication
Get your API key from scrapebadger.com and pass it to the client:
from scrapebadger import ScrapeBadger
client = ScrapeBadger(api_key="sb_live_xxxxxxxxxxxxx")
You can also set the SCRAPEBADGER_API_KEY environment variable:
export SCRAPEBADGER_API_KEY="sb_live_xxxxxxxxxxxxx"
Available APIs
| API | Description | Documentation |
|---|---|---|
| Web Scraping | Scrape any website with JS rendering, anti-bot bypass, and AI extraction | Web Scraping Guide |
| 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | Twitter Guide | |
| 19 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products | Google Guide | |
| Vinted | Search items, item details, user profiles, brands, colors, statuses, and markets | Vinted Guide |
| Search posts, subreddits, users, and domains; fetch post comments, subreddit rules, moderators, wiki pages, user trophies | Reddit Guide | |
| User profile/about/related/posts/videos/reels/tagged/pinned/followers/following/stories/highlights, media detail/comments/replies/likers/oEmbed, search (users/hashtags/places/top/reels/music/autocomplete), hashtag/location/audio feeds | Instagram Guide | |
| Amazon | 14 endpoints — search, autocomplete, product detail, offers, reviews, bestsellers, new releases, deals, category browse, seller profile/products/feedback, markets, categories | Amazon Guide |
| eBay | 13 endpoints across 18 markets — search, search by image (visual search), completed/sold search, item detail, item reviews, seller profile/items/feedback, category browse, categories, autocomplete, markets | eBay Guide |
| YouTube | 39 endpoints — search, autocomplete, video detail/related/comments/replies/transcript/captions/streams/live-chat/batch, channel detail + videos/shorts/streams/playlists/community/about/subscriber-count/in-channel-search/resolve, playlists/items/mixes, trending/hashtag/home, shorts, community post/comments, music search, oembed, categories/languages/regions | YouTube Guide |
| TikTok | 25 endpoints — user profile/videos/followers/following/liked/reposts, video detail/comments/replies/related/transcript/oEmbed, search (general/videos/hashtags/users), music detail/videos, hashtag detail/videos, trending videos/hashtags/songs, ad library, regions | TikTok Guide |
| Immobiliare | 8 endpoints across immobiliare.it, indomio.es, indomio.gr, immotop.lu — autocomplete, search, listing detail, agency profile/listings, price stats, markets, reference | Immobiliare Guide |
| LoopNet | 5 endpoints across loopnet.com/.ca/.co.uk/.fr/.es — commercial-real-estate search (for-lease/for-sale/auctions), listing detail, broker profile, markets, property types | LoopNet Guide |
| Apartments.com | US rental listings with UNIT-LEVEL pricing — search by location with bed/price filters, plus per-unit rent, beds, baths, sqft and availability for every rentable unit | Apartments Guide |
| Walmart | 11 endpoints (US-only) — search, category browse, deals feed, autocomplete, product detail, reviews, seller profile/products, store detail, markets | Walmart Guide |
| Baidu | 4 endpoints — web search (language + date filters), news vertical, image search, autocomplete. Results carry the real target URL, not just Baidu's tracking redirect | Baidu Docs |
| Bing | 6 endpoints — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | Bing Docs |
| DuckDuckGo | 7 endpoints — web search (with abstract box), image/news/video search, autocomplete, instant answers, regions | DuckDuckGo Docs |
| Yahoo | 6 endpoints across 35 markets — web search (with ads + related searches), image search, video search, news vertical, autocomplete, markets | Yahoo Docs |
| Yandex | 4 endpoints across 6 markets (tr/com/ru/by/kz/uz) — web search (organic + ads + sitelinks + inline media), image search, reverse-image (CBIR) search, markets | Yandex Docs |
| ChatGPT | Prompt the real chatgpt.com anonymously — structured answer with citations anchored to character offsets, the full retrieved search trail, and AEO/GEO brand-visibility analysis | ChatGPT Guide |
| Gemini | Prompt the real gemini.google.com anonymously — structured answer with cited web sources, the full retrieved search trail, and AEO/GEO brand-visibility analysis | Gemini Docs |
Error Handling
from scrapebadger import (
ScrapeBadger,
ScrapeBadgerError,
AuthenticationError,
RateLimitError,
InsufficientCreditsError,
NotFoundError,
ValidationError,
ServerError,
)
async with ScrapeBadger(api_key="your-key") as client:
try:
user = await client.twitter.users.get_by_username("elonmusk")
except AuthenticationError:
print("Invalid API key")
except RateLimitError as e:
print(f"Rate limited. Retry after {e.retry_after} seconds")
print(f"Limit: {e.limit}, Remaining: {e.remaining}")
except InsufficientCreditsError:
print("Out of credits! Purchase more at scrapebadger.com")
except NotFoundError:
print("User not found")
except ValidationError as e:
print(f"Invalid parameters: {e}")
except ServerError:
print("Server error, try again later")
except ScrapeBadgerError as e:
print(f"API error: {e}")
Configuration
Custom Timeout and Retries
from scrapebadger import ScrapeBadger
client = ScrapeBadger(
api_key="your-key",
timeout=120.0, # Request timeout in seconds (default: 300)
max_retries=5, # Retry attempts (default: 10)
)
Advanced Configuration
from scrapebadger import ScrapeBadger
from scrapebadger._internal import ClientConfig
config = ClientConfig(
api_key="your-key",
base_url="https://scrapebadger.com",
timeout=300.0,
connect_timeout=10.0,
max_retries=10,
retry_on_status=(500, 502, 503, 504),
headers={"X-Custom-Header": "value"},
)
client = ScrapeBadger(config=config)
Retry Behavior
The SDK automatically retries requests that fail with 500, 502, 503, or 504 status
codes, as well as transport-level failures (timeouts, network errors, dropped
connections), using exponential backoff (1s, 2s, 4s, 8s, ...). Retryable HTTP
responses with a valid Retry-After header wait at least that long, whether
given as seconds or an HTTP-date, without shortening exponential backoff.
Missing or invalid headers keep the backoff, and max_retries still limits
retry attempts. Each retry logs a warning:
⚠ 503 Service Unavailable — retrying in 4s (attempt 3/10)
To see these warnings, configure Python logging:
import logging
logging.basicConfig(level=logging.WARNING)
Rate Limit Aware Pagination
When using *_all pagination methods, the SDK reads X-RateLimit-Remaining and
X-RateLimit-Reset headers from each response. When remaining requests drop below
20% of your tier's limit, pagination automatically slows down to spread requests
across the remaining window — preventing 429 errors. A warning is logged when
throttling activates:
⚠ Rate limit: 25/300 remaining (resets in 42s), throttling pagination to ~0.6 req/s
This works transparently with all tier levels (Free: 60/min, Basic: 300/min, Pro: 1000/min, Enterprise: 5000/min).
Development
Setup
# Clone the repository
git clone https://github.com/scrape-badger/scrapebadger-python.git
cd scrapebadger-python
# Install dependencies with uv
uv sync --dev
# Install pre-commit hooks
uv run pre-commit install
Running Tests
# Run all tests
uv run pytest
# Run with coverage
uv run pytest --cov=src/scrapebadger --cov-report=html
# Run specific tests
uv run pytest tests/test_client.py -v
Code Quality
# Lint
uv run ruff check src/ tests/
# Format
uv run ruff format src/ tests/
# Type check
uv run mypy src/
# All checks
uv run ruff check src/ tests/ && uv run ruff format --check src/ tests/ && uv run mypy src/
Contributing
Contributions are welcome! Please read our Contributing Guide for details.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run tests and linting (
uv run pytest && uv run ruff check) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
License
This project is licensed under the MIT License - see the LICENSE file for details.
Support
- Documentation: docs.scrapebadger.com
- Issues: GitHub Issues
- Email: support@scrapebadger.com
- Discord: Join our community
Made with ❤️ by ScrapeBadger
1 maintainer
Metadata
1 maintainer
Release files for scrapebadger 0.54.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapebadger-0.54.0.tar.gz | 214.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapebadger-0.54.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 558.2 kB
Release files / scrapebadger-0.54.0.tar.gz
| Download URL | scrapebadger-0.54.0.tar.gz |
|---|---|
| Size | 214.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dfd21524a64f82a6e7fddde189365a9e9ae8f6cbe4616e9b182ceb9ff7989db4
|
|
BLAKE2b-256 checksum How to use checksums |
523f816e6097f14eac850b5d75e8570c5cacd573d9905345e78eedcfd4cbabcf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 5, 2026.
Transparency logRelease files / scrapebadger-0.54.0-py3-none-any.whl
| Download URL | scrapebadger-0.54.0-py3-none-any.whl |
|---|---|
| Size | 343.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
09647bdbfd506e363fe0e1067eb6590f58926e135a81ce900eb9a415cf184b64
|
|
BLAKE2b-256 checksum How to use checksums |
ca13b0061dae9de911ade6b1ac92cb76d440cb9f44275e6d3e63631659ab6ae9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 5, 2026.
Transparency log
