litescrape-sdk

Python SDK for the Litescrape API: validated, concurrent, retrying calls to all 139 endpoints, covering search engines (Google, Bing, Brave, DuckDuckGo, Yahoo, Yandex, Baidu, Naver), AI answers (Google AI Mode and Gemini, Brave AI Mode, Microsoft Copilot), maps and reviews, app stores, social platforms (YouTube, TikTok, Instagram, Facebook, X), retail, jobs, real estate, travel, and web pages, with results returned in input order. The endpoint table lists every request class.

pip install litescrape-sdk

import os
from litescrape_sdk import GoogleMaps, scrape

os.environ["LITESCRAPE_API_KEY"] = "ls_live_..."

results = scrape(
    [
        {"endpoint": "google_search", "q": "coffee grinders", "gl": "us"},
        GoogleMaps(q="coffee", type="search", ll="@40.745,-73.988,14z"),
    ]
)
for result in results:
    print(result.ok, result.data or result.error)

help(litescrape_sdk.scrape) documents every argument, the retry and concurrency rules, and the error types. ascrape is the same function for asyncio code, and litescrape_sdk.REQUEST_TYPES maps each endpoint slug to its request class.

Most endpoints are sent as GET with query parameters. Array parameters on the job search endpoints (IndeedSearch(employer_keys=[...]), LevelsSearch(companies=[...]), ...) are JSON-encoded in the query for you. Four endpoints take arrays or nested objects and are sent as a JSON POST body instead: WebFetch, WebScreenshot, GoogleShoppingMerchantUrls, and CopilotChatCompletions. Each class's method says which, and body() shows the JSON it sends. eBay's underscore parameters are fields without the underscore: EbaySearch(nkw="lego", ipg=25) sends _nkw and _ipg.

Development: uv sync, then uv run pytest, uv run ruff check ., and uv run ruff format ..

Durable batches and recovery

Requires an API deployment with durable batch support. Set batched=True to submit every query before polling for completion. The server continues processing accepted jobs if your computer disconnects or the Python process exits. Responses are retained for 24 hours after completion.

from litescrape_sdk import GoogleSearch, scrape

queries = [GoogleSearch(q="coffee grinders"), GoogleSearch(q="espresso machines")]
results = scrape(queries, batched=True, cache_path="./my-batch.sqlite3")

# After a crash or disconnect, rerun with the same inputs, key and cache path:
results = scrape(queries, batched=True, useCache=True, cache_path="./my-batch.sqlite3")
for result in results:
    print(result.job_id, result.data or result.error)

The SDK saves the entire workload's submission intentions and batch IDs before sending its first request, then saves returned job IDs. The server provisions dedicated batch workers from accepted inputs and assigns work as matching identities become ready. useCache=False is the default and creates fresh jobs. useCache=True retrieves matching saved jobs; it creates replacements only when the server confirms a job is missing or expired. A polling outage does not trigger a fresh charge. Reordering distinct queries preserves their cache matches; identical queries are matched by occurrence. Keep separate cache files for independently resumable batches. The default path is ~/.cache/litescrape/jobs.sqlite3, configurable with LITESCRAPE_JOB_CACHE. The cache stores batch IDs, job IDs and request fingerprints, without raw API keys or query text.

New jobs reserve one credit; terminal failures refund that reservation. Polling and retrying the same submission do not consume more credits. Batch mode skips the synchronous whole-list balance check so previously paid results can be recovered even with no credits remaining. Check every Result for submission errors, including insufficient credits for new jobs.

concurrency controls simultaneous submission/poll requests (default 32); execution uses separate server batch capacity. Network retries and polling use exponential backoff with jitter, capped at 30 seconds; retries honor Retry-After. All results are returned in input order. ascrape accepts the same options. A server request_timeout applies to each worker attempt after queueing.

Google Search fast mode

Google Search supports GoogleSearch(q="coffee grinders", fast_mode=True) to return only organic results plus search metadata and parameters. This skips AI Overview and other result groups. The default is the full response; fast mode is unavailable on the dedicated AI Overview endpoint. Requires an API deployment that supports fast_mode.

num on GoogleSearch and GoogleAiOverview accepts 1 to 10, the most Google returns on one page. To page further, send GoogleSearch(page_token=...) alone with the pagination.next_page_token from the previous response, within five minutes; start also still works.

Fetch and screenshots (Alpha)

WebFetch renders a public web page and returns it as Markdown, HTML, or plain text; WebScreenshot returns a PNG screenshot encoded as base64. Both are sent as a JSON body, so list selectors, cookies, scripts, and a Viewport work directly.

import base64

from litescrape_sdk import WebFetch, WebScreenshot, scrape

page, shot = scrape(
    [
        WebFetch(url="https://example.com", respond_with="markdown", target_selector=["article", "main"]),
        WebScreenshot(url="https://example.com", full_page=True, viewport={"width": 1440, "height": 900}),
    ]
)
page = page.raise_for_error()
print(page["title"], page["url"])
print(page["content"])
png = base64.b64decode(shot.raise_for_error()["screenshot_base64"])

A rendered error page is still a successful, billed capture unless you set assert_status_code (for example 200); a different origin status is then rejected without billing. respond_with is content (the default), markdown, html, text, frontmatter, or a + combination such as markdown+frontmatter. Screenshots moved from respond_with="screenshot" to WebScreenshot. The 0.6 options wait_until, with_links, and with_images are replaced by respond_timing, retain_links, and retain_images.

Copilot and merchant URLs

from litescrape_sdk import CopilotChatCompletions, GoogleShoppingMerchantUrls, scrape

results = scrape(
    [
        CopilotChatCompletions(
            messages=[{"role": "user", "content": "What is the weather near me right now?"}],
            country="US",
            state="CA",
        ),
        GoogleShoppingMerchantUrls(
            products=[{"headline_offer_docid": "5148952067814683768", "merchant": "Target"}]
        ),
    ]
)
answer = results[0].raise_for_error()
print(answer["choices"][0]["message"]["content"])

Each Copilot call starts a fresh conversation; resend earlier messages for context. The last message must have role user. Merchant URL lookups take 1 to 100 headline offers copied from a GoogleShopping result.

Request deadlines

Set request_timeout on scrape or ascrape to apply a server deadline to every item. Set timeout on an individual request to override that default. Both accept seconds greater than 0 and at most 90, including fractional seconds; omitting them preserves the API's standard behavior.

from litescrape_sdk import GoogleSearch, RequestDeadlineExceededError, scrape

(result,) = scrape(
    [GoogleSearch(q="coffee grinders")],
    request_timeout=15,
    attempts=1,
)
if isinstance(result.error, RequestDeadlineExceededError):
    print(result.error.request_id, result.error.retryable, result.error.message)
else:
    print(result.raise_for_error())

# A per-item deadline also works in dictionaries.
results = scrape(
    [
        GoogleSearch(q="coffee", timeout=10),
        {"endpoint": "google_search", "q": "tea", "timeout": 20},
    ],
    attempts=1,
)

The deadline covers the entire API request from gateway receipt, including admission, scraping, and billing. On expiry, the API returns HTTP 503 with the new request_deadline_exceeded code, retryable: true, a request ID, and a description. That attempt is not charged. The API cancels the underlying scrape to release concurrency; credit and concurrency cleanup can finish shortly after the response.

The SDK retries this response using its usual attempts and Retry-After rules. Each attempt gets its own deadline; use attempts=1 as above to receive the first timeout immediately. SDK queueing, the status check, network transit, retry waits, and the full batch are outside the server deadline.

The existing scrape(..., timeout=120) argument remains the HTTP transport timeout. Keep it longer than the server deadline so the API can return its error. A client-side TransportError alone does not guarantee that the request was unbilled.

Store APIs

The nine Google Play and Apple App Store operations (GooglePlayApps through AppleAppStoreReviews in the endpoint table) use your existing key. Fields depend on what the storefront supplies.

from litescrape_sdk import GooglePlayApps, GooglePlayProduct, AppleAppStoreSearch, scrape

results = scrape(
    [
        GooglePlayApps(q="coffee", hl="en", gl="us"),
        GooglePlayProduct(product_id="com.duolingo"),
        AppleAppStoreSearch(term="coffee", country="us", num=10),
    ]
)
for result in results:
    print(result.raise_for_error())

Follow litescrape_pagination.next or use the returned continuation with the same operation and parameters. Each successful page consumes one call; failures do not. For Google Play, chart, next_page_token, section_page_token, and see_more_token are mutually exclusive. Queries exclude category filters and charts. Search text is limited to 2,048 UTF-8 bytes, so non-ASCII characters can consume more than one byte. Apple search also limits URL-encoded terms to 4,096 bytes. Apps/games charts require store_device="phone" or an omitted device; other device storefronts are browsable without a chart. Omit store_device when using an apps or games query or category. GooglePlayGames(q=...) uses the shared Android app search; omit q or choose games_category to browse games. Apple review pages are one-based; exhausted pages return an empty list. Mac reviews use newest-first ordering.

search_metadata.raw_file and prettify_file, when returned, link to authenticated response artifacts retained for at least seven days (today and the previous seven UTC date buckets). Download them with the same bearer key; downloads are unbilled. Apple search applies category and case-insensitive developer-name filters before the num ceiling. The native search window can contain fewer matching results than that ceiling. See the API reference for every parameter and response group.

Endpoints

Alpha endpoints are newer; their response fields can change as the sources do.