Skip to main content

WebCrawler API Python SDK

A Python SDK for interacting with the WebCrawlerAPI.

Get your API key at dash.webcrawlerapi.com/access.

Installation

pip install webcrawlerapi

Crawling

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="your_api_key")

job = client.crawl(
    url="https://example.com",
    output_formats=["markdown"],
    items_limit=10,
)

for item in job.job_items:
    if item.status == "done":
        print(item.original_url)
        print(item.content)

Scraping

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="your_api_key")

result = client.scrape(
    url="https://example.com",
    output_formats=["markdown"],
)

if result.success:
    print(result.markdown)
else:
    print(f"Error: {result.error_code} — {result.error_message}")

Batch scrape

batch = client.scrape(
    ["https://example.com/", "https://example.com/pricing"],
    output_formats=["markdown"],
    max_concurrency=5,
)

print(batch.status, batch.completed, batch.failed)
for item in batch.data:
    print(item.url, item.status, item.markdown_url)

Passing a list to scrape() creates a batch and polls until it finishes (a single URL string still returns a single result). Batch-only options: max_concurrency, ignore_invalid_urls, timeout_seconds. For manual control use scrape_batch_async(), get_batch(id, limit=, cursor=), get_batch_errors(id) and cancel_batch(id).

Documentation

Requirements

  • Python 3.6+
  • requests >= 2.25.0

License

MIT License

Metadata

Release files for webcrawlerapi 2.1.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webcrawlerapi 2.1.8
File Size Uploaded Provenance Actions
webcrawlerapi-2.1.8.tar.gz 18.7 kB Without provenance Details for webcrawlerapi-2.1.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for webcrawlerapi 2.1.8
File Interpreter ABI Platform Actions
webcrawlerapi-2.1.8-py3-none-any.whl Python 3 none any Details for webcrawlerapi-2.1.8-py3-none-any.whl

Total release size: 37.8 kB

Release files / webcrawlerapi-2.1.8.tar.gz

Download URL webcrawlerapi-2.1.8.tar.gz
Size 18.7 kB
Tags Source
SHA-256 checksum
How to use checksums
4a65c4ea2b7183c367e90d7f8f753a8a1c821f87e99b22cbeccaf6dd16c892eb
BLAKE2b-256 checksum
How to use checksums
86a8a4191035d1eed81037bb5ffe8493a1fcae770612fe72097a3770f81154dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Release files / webcrawlerapi-2.1.8-py3-none-any.whl

Download URL webcrawlerapi-2.1.8-py3-none-any.whl
Size 19.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
139fb5be9d2a95543e12cd01cfcf7a2e5e50d755bddb4ac3a41a450b61924e42
BLAKE2b-256 checksum
How to use checksums
90a9ed466825b8d5d341d533b47d917ef329a756bb66e5c1588f375f71e50eab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.15.1

Security for webcrawlerapi 2.1.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page