webscraping_ai

Official Python client for the WebScraping.AI API — web scraping with Chromium JavaScript rendering, rotating datacenter/residential/stealth proxies, and AI-powered question answering and structured field extraction on any page. Sync and async clients included. See the API documentation for the full parameter reference.

4.0 is a hard break from 3.x. See CHANGELOG.md for the migration notes. If you cannot update your call sites yet, stay on webscraping_ai == 3.2.1.

Install

pip install webscraping_ai

Requires Python 3.9 or newer.

Quick start

Sign up to get an API key — a free trial, no credit card required. Your key lives in the dashboard.

from webscraping_ai import Client

client = Client(api_key="YOUR_API_KEY")

# Page HTML
html = client.html("https://example.com")

# Visible text, optionally as a structured JSON response
text = client.text("https://example.com", text_format="json", return_links=True)

# CSS-selected HTML
heading = client.selected("https://example.com", selector="h1")
multiple = client.selected_multiple("https://example.com", selectors=["h1", "p"])

# LLM-powered helpers
answer = client.question("https://example.com", question="What is the page title?")
fields = client.fields(
    "https://example.com",
    fields={"title": "Main product title", "price": "Current product price"},
)

# Google search results (SERP) for a query
results = client.serp("coffee machines", gl="us", hl="en", page=1)

# Search from a fixed location, two results pages in one call
local = client.serp("coffee machines", location="Austin, TX", pages=2)

# Look up names for `location`
austin = client.serp_locations("austin")[0]["canonical_name"]  # "Austin,Texas,United States"

# Structured data for a page on a supported site (YouTube, TikTok, X, LinkedIn, Instagram, Reddit, ...)
video = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(video["data"]["title"])

# Account quota
info = client.account()

The client is also a context manager, which closes the underlying connection pool on exit:

with Client(api_key="...") as client:
    client.html("https://example.com")

Async usage

AsyncClient mirrors Client but uses async def methods backed by httpx.AsyncClient:

import asyncio
from webscraping_ai import AsyncClient

async def main():
    async with AsyncClient(api_key="YOUR_API_KEY") as client:
        html = await client.html("https://example.com")
        print(html)

asyncio.run(main())

Error handling

Every non-2xx response is mapped to a typed exception so you can except on the situation you actually care about rather than parsing status codes:

from webscraping_ai import (
    Client,
    AuthenticationError,
    RateLimitError,
    PaymentRequiredError,
    APITimeoutError,
    APIConnectionError,
)

client = Client(api_key="YOUR_API_KEY")

try:
    client.html("https://example.com")
except AuthenticationError:
    ...  # 403 — wrong or missing API key
except PaymentRequiredError:
    ...  # 402 — out of credits
except RateLimitError:
    ...  # 429 — too many concurrent requests
except APITimeoutError:
    ...  # request did not complete in time
except APIConnectionError:
    ...  # transport-level failure

All exceptions inherit from WebScrapingAIError, so you can catch everything the client raises with a single except if you prefer. API errors expose the parsed error envelope (message, status, status_code, status_message, body, response_body).

APITimeoutError and APIConnectionError are raised without chaining the underlying httpx exception (it holds the request URL, which contains your API key); the original exception type is named in the message instead.

Logging and your API key

The API key travels in the query string, and httpx logs every request URL at INFO on the httpx logger. Importing webscraping_ai installs a logging.Filter on that logger that rewrites api_key=<value> to api_key=[REDACTED], so enabling INFO logging does not leak the key. The filter only covers the httpx logger; if you log request URLs yourself, redact them too.

Endpoint reference

Every page-fetch method accepts the full set of API parameters as keyword arguments: headers, timeout, js, js_timeout, wait_for, proxy, country, custom_proxy, device, error_on_404, error_on_redirect, js_script, plus the per-endpoint extras (return_script_result, format, text_format, return_links, selector, selectors, question, fields). See the API documentation for the full parameter reference.

SERP

client.serp(q, *, engine=None, gl=None, hl=None, page=None, pages=None, location=None, uule=None, lat=None, lon=None, radius=None) returns parsed search engine results for a query. It is query-shaped rather than URL-shaped, so none of the page-fetch parameters above apply. Billed per results page returned (see pricing); failed searches are not charged. Raises ValueError before any request when q is not a non-blank str, page, pages or radius is not an int >= 1, or lat/lon is not a finite number (the server also rejects these with a 400, not billed; checking client-side saves the round trip). q and location are sent as given.

Every search runs from one fixed location: location, lat + lon or uule, otherwise the capital of the gl country, so repeats and later pages are consistent. gl defaults to the location's country. Use only one of location, uule or lat/lon; an unknown location or a bad combination gets a 400 from the server (not billed). lat/lon are sent as plain decimals rounded to 7 places. pages fetches consecutive pages as one de-duplicated list, billed per page returned.

results = client.serp("coffee machines", gl="gb", page=2)
print(results["search_information"]["organic_results_state"])  # "Results for exact spelling"
for r in results["organic_results"]:
    print(r["position"], r["title"], r["link"])
print(results["pagination"])  # {"current": 2, "next": 3}

local = client.serp("coffee machines", location="Austin, TX", pages=2)
print(local["search_parameters"]["location_used"])  # "Austin,Texas,United States"
for r in local["organic_results"]:
    print(r["page"], r["position"], r["title"])  # position counts 1..N across both pages
print(local["pagination"]["pages_returned"])  # 2

The response dict has search_parameters (engine, q, gl, hl, page, plus pages when > 1 and location_requested, location_used, lat, lon, radius when relevant), search_information (query_displayed, organic_results_state, optional showing_results_for and total_results), organic_results (position, title, link, domain, displayed_link, optional snippet and date, and page when pages > 1), optional related_searches (query), and pagination (current, optional next, and pages_returned when pages > 1). position is 1-based within the page, or counts through the combined list when pages > 1. Optional keys are absent when the engine does not show them, so use .get() for those.

SERP locations

client.serp_locations(q=None, *, limit=None) looks up names for serp's location. It is free and needs no API key (the client still sends it). Without q the most populous locations are returned; limit caps the count (the server allows up to 10). Raises ValueError before any request when limit is not an int >= 1. q matches English names (e.g. "petersburg"); serp's location also accepts local names for large cities.

locations = client.serp_locations("austin", limit=3)
best = locations[0]
print(best["canonical_name"], best["target_type"], best["gps"])
# Austin,Texas,United States City [-97.7431, 30.2672]   (gps is [longitude, latitude])
results = client.serp("coffee machines", location=best["canonical_name"])  # or best["id"]

Each entry has id (a str), google_id, google_parent_id (may be None), name, canonical_name, country_code, target_type (e.g. City, State, Country; an open set), reach (approximate population) and gps.

Structured data

client.data(url, *, country=None, transcript=None, transcript_language=None, **params) (and await AsyncClient.data(...)) returns structured JSON for a public page on a supported site. Pass the page's normal URL; the site (provider) and page kind (type) are detected from it. Priced per site (see pricing), including pages that parse empty (parse_status "parse_failed") or no longer exist ("not_found"); unsupported URLs and failed fetches are not charged. None of the page-fetch parameters above apply.

Supported sites today include, for example, YouTube (video/channel/playlist), TikTok (video/profile), X/Twitter (tweet/profile), LinkedIn (company/job/profile), Instagram (post/reel/profile) and Reddit (post/subreddit/user). More sites are added server-side and work with this package without an upgrade: the client never checks the URL's site. An unsupported URL or page type returns a 400 that is not charged (BadRequestError). Its message lists what is supported.

result = client.data("https://www.youtube.com/watch?v=dQw4w9WgXcQ", transcript=True)
result["request_parameters"]  # {"url": "...", "provider": "youtube", "type": "video"}
result["parse_status"]        # "ok" (or "parse_failed" / "not_found")
result["data"]                # shape depends on provider and type; may be None

try:
    client.data("https://example.com/")
except BadRequestError as e:
    print(e.message)  # "Unsupported URL for /data. Supported sites: youtube, tiktok, ..."

provider, type and parse_status are open sets of strings, and data is the decoded JSON as-is (no per-site classes), so new sites and fields show up without a package release.

API response-shape notes

Two endpoints return shapes that differ from the OpenAPI spec examples. The client returns the raw response unchanged, so:

• /ai/fields wraps the extracted fields under a result key: {"result": {"title": "...", "price": "..."}}.
• /selected-multiple returns list[list[str]], not a flat list[str].

Development

mise install                    # or use python 3.13 from any source
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .
mypy src/webscraping_ai

Smoke testing

bin/smoke.py hits every endpoint once against the live API through the sync Client, plus one account call through AsyncClient. It puts src/ first on sys.path, so it always tests the working tree (you still need the runtime deps, e.g. from pip install -e ".[dev]"). It is not part of the pytest suite. The four page calls run with js=False and proxy="datacenter" to keep the sweep cheap; the two SERP calls (one plain, one with location="Austin, TX", pages=2) are billed per results page returned, /data is one YouTube video, and serp_locations and account are free. A second /data call on https://example.com/ must come back as the server's free 400, proving there is no client-side site filter. Each case checks the result shape, not just that no exception was raised (SERP must return organic results for the query sent, the located SERP must report an Austin location_used, serp_locations("austin") must put Austin first, /data must parse a title, selected_multiple must match something, and so on), and FAIL lines redact the API key.

WEBSCRAPING_AI_API_KEY=... python bin/smoke.py

Each call prints an ok or FAIL line (any exception counts as a failure, and the sweep continues); the script exits non-zero if any call fails.

Links

• WebScraping.AI — features, pricing, signup
• API documentation
• Dashboard — API key, usage, request builder
• Other official clients: JavaScript · Ruby · PHP · Go · Java · .NET · CLI · MCP server · n8n node
• Support: support@webscraping.ai

License

MIT.