crawlora 1.5.0.dev2

Official Python SDK for the Crawlora web-scraping API: typed grouped and dynamic operation calls for every public endpoint, with retries, pagination, hooks, and an async client.

Crawlora Python SDK

Python client for the public Crawlora API. Use it to call Crawlora scraping, search, marketplace, media, maps, finance, and usage endpoints with generated type stubs for editor and type-checker support.

• Runtime: Python 3.10+
• Auth: x-api-key
• Default API base URL: https://api.crawlora.net/api/v1
• Reference: operations and recipes

Install

Published on PyPI. The current release is a prerelease (1.5.0.dev1), so install it with --pre:

pip install --pre crawlora

(Git installs from beta tags also work, e.g. pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@latest".)

API Key

Create or sign in to your Crawlora account at crawlora.net, then create an API key in the dashboard.

read -r CRAWLORA_API_KEY
export CRAWLORA_API_KEY

First Request

import os
from crawlora import CrawloraClient

crawlora = CrawloraClient(api_key=os.environ["CRAWLORA_API_KEY"])

response = crawlora.bing.search(
    q="coffee shops",
    count=10,
)

print(response["data"]["results"][0])

Endpoint groups are generated from the public API contract, so common calls are available as methods such as crawlora.bing.search(...), crawlora.youtube.transcript(...), and crawlora.google.map_search(...).

Typed Dynamic Calls

You can also call by operation id. Literal operation ids are covered by the generated .pyi stubs, so type checkers can infer the matching parameter and response aliases:

response = crawlora.request("bing-search", {
    "q": "coffee shops",
    "count": 10,
})

Generated stubs include operation ids, endpoint groups, keyword parameters, enum values, response aliases, and reserved request options.

Configuration

crawlora = CrawloraClient(
    api_key=os.environ["CRAWLORA_API_KEY"],
    base_url="https://api.crawlora.net/api/v1",
    timeout=30,
    retries=2,
    retry_delay=0.25,
    headers={"x-client": "my-app"},
)

Per-request options are available through reserved keyword arguments. Header names are matched case-insensitively, so request headers can override default auth, user-agent, and content headers without duplicating variants such as x-api-key and X-API-KEY:

response = crawlora.bing.search(
    q="coffee shops",
    _timeout=10,
    _headers={"x-request-id": "search-001"},
)

Text Responses

Most endpoints return JSON. _response_type must be auto, json, or text. Endpoints that support alternate text output, such as YouTube transcripts, can opt into text mode:

transcript = crawlora.youtube.transcript(
    id="VIDEO_ID",
    format="text",
    _response_type="text",
)

print(transcript)

Errors

Failed API calls raise CrawloraError:

from crawlora import CrawloraError

try:
    crawlora.bing.search(q="coffee shops")
except CrawloraError as error:
    print(error.status, error.code, error.body)
    raise

The error includes status, optional API code, parsed body, raw_body, response headers, and the underlying parser or transport exception as __cause__ when available. Retryable responses honor positive Retry-After headers, capped at 30 seconds. Timeout-like transport failures use the Crawlora request timed out SDK message.

CrawloraError has three subclasses for branching on the failure kind: CrawloraClientError (4xx, request rejected), CrawloraServerError (5xx), and CrawloraNetworkError (transport failure or timeout before a response).

Async

AsyncCrawloraClient mirrors the synchronous client for asyncio applications:

from crawlora import AsyncCrawloraClient

crawlora = AsyncCrawloraClient(api_key="YOUR_API_KEY")
result = await crawlora.bing.search(q="coffee shops")

It reuses the same validation, retries, and Retry-After handling, running each request in a worker thread so the package stays dependency-free.

Pagination

client.paginate yields successive pages, advancing the page/offset query parameter and stopping when a page returns no data:

for page in crawlora.paginate("ebay-seller-feedback", {"seller": "acme"}):
    for review in page["data"]:
        print(review)

AsyncCrawloraClient.paginate is the async for equivalent. Override detection with page_param, start, step, and max_pages.

Examples

Runnable examples live under examples/ and skip cleanly when required environment variables are missing:

python3 examples/bing_search.py
python3 examples/youtube_transcript.py

Set CRAWLORA_BASE_URL to point examples at a staging or local API.

Package Notes

The import name is crawlora:

from crawlora import CrawloraClient

The future PyPI package target is also crawlora, but registry publication is not enabled yet. Until then, install from an explicit Git beta tag or the moving latest tag as shown above.