webcrawlerapi 2.1.2

Python SDK for WebCrawler API

WebCrawler API Python SDK

A Python SDK for interacting with the WebCrawlerAPI.

Get your API key at dash.webcrawlerapi.com/access.

Installation

pip install webcrawlerapi

Crawling

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="your_api_key")

job = client.crawl(
    url="https://example.com",
    output_formats=["markdown"],
    items_limit=10,
)

for item in job.job_items:
    if item.status == "done":
        print(item.original_url)
        print(item.content)

Scraping

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="your_api_key")

result = client.scrape(
    url="https://example.com",
    output_formats=["markdown"],
)

if result.success:
    print(result.markdown)
else:
    print(f"Error: {result.error_code} — {result.error_message}")

Documentation

• Crawling — parameters, async, job/item responses, cancellation
• Scraping — parameters, async, response fields
• Advanced — output formats, structured outputs, webhooks, S3 actions

Requirements

• Python 3.6+
• requests >= 2.25.0

License

MIT License