Real browsers. Structured extraction. Screenshots and automation.

Python SDK

Turn a URL into rendered HTML, readable Markdown, structured data, or a PNG screenshot. Connect a remote browser with Playwright when your workflow needs interaction.

Website · API documentation · Get an API key · 中文

• Python 3.9+ with synchronous and asynchronous clients.
• Zero runtime dependencies; uses the Python standard library.
• Explicit request timeouts, response validation, and typed errors.
• No automatic retries of metered requests or browser creation.

Install

pip install adscrawl

Your first request

Create an account, create an API key, and set ADSCRAWL_API_KEY in the server environment.

from adscrawl import AdsCrawl

client = AdsCrawl()
markdown = client.markdown({
    "url": "https://www.adscrawl.net",
    "waitUntil": "domcontentloaded",
})
print(markdown)

You can also use AdsCrawl(api_key="..."). Keep API keys in server-side code. Request dictionaries use the same camelCase fields as the HTTP API and TypeScript SDK.

Rendered content and screenshots

from pathlib import Path
from adscrawl import AdsCrawl

client = AdsCrawl()
html = client.html({"url": "https://www.adscrawl.net"})
article = client.article({"url": "https://www.adscrawl.net"})
print(article["title"], article["textContent"])

png = client.screenshot({
    "url": "https://www.adscrawl.net",
    "viewport": {"width": 1440, "height": 900},
    "fullPage": True,
    "waitUntil": "load",
})
Path("page.png").write_bytes(png)

html() returns HTML by default. markdown() returns Markdown, article() returns a dictionary, and screenshot() returns PNG bytes. Use a custom proxy or managed countryCode; they cannot be combined.

Proxy and fingerprint: BrowserScan screenshot

AdsCrawl uses real browsers with configurable routing and browser fingerprints. This example uses managed GLOBAL routing unless proxy environment variables are present, opens BrowserScan, and saves the returned screenshot. The same browser workflow can access and render Pixelscan and IPhey; inspect the returned page for its current score.

import os
from pathlib import Path
from adscrawl import AdsCrawl

client = AdsCrawl()
server = os.getenv("ADSCRAWL_PROXY_SERVER")
username = os.getenv("ADSCRAWL_PROXY_USERNAME")
password = os.getenv("ADSCRAWL_PROXY_PASSWORD")
if bool(username) != bool(password):
    raise RuntimeError("Set both proxy username and password, or neither.")
if not server and (username or password):
    raise RuntimeError("Set ADSCRAWL_PROXY_SERVER with proxy credentials.")

routing = {"countryCode": "GLOBAL"}
if server:
    proxy = {"server": server}
    if username and password:
        proxy.update({"username": username, "password": password})
    routing = {"proxy": proxy}

png = client.screenshot({
    "url": "https://www.browserscan.net/",
    **routing,
    "viewport": {"width": 1440, "height": 900},
    "fullPage": True,
    "waitUntil": "networkidle",
    "timeoutMs": 60_000,
    "userAgentMode": "random",
    "userAgentOs": "windows",
    "fingerprint": {
        "webRtc": "forward", "webGl": "random", "webGpu": "random",
        "webGlImage": "random", "canvas": "random",
        "audioContext": "random", "clientRects": "random",
        "speechVoices": "random", "fonts": "random",
        "hardware": "random", "doNotTrack": "random",
    },
}, timeout_ms=75_000)
Path("browserscan.png").write_bytes(png)

Structured extraction

from adscrawl import AdsCrawl

client = AdsCrawl()
print(client.spa.templates()["templates"])

result = client.spa.extract({
    "url": "https://www.adscrawl.net",
    "fields": {
        "title": {"source": "dom", "selector": "h1", "value": "text", "required": True},
    },
})
print(result["data"]["title"], result["missingFields"])

inspection = client.spa.inspect({"url": "https://www.adscrawl.net"})
print(inspection["candidates"])

Remote browsers with Playwright

pip install "adscrawl[playwright]"
playwright install chromium

from adscrawl import AdsCrawl
from playwright.sync_api import sync_playwright

client = AdsCrawl()
session = client.cdp.create({
    "idleTimeoutMs": 600_000,
    "maxSessionMs": 3_600_000,
    "browserSettings": {"viewport": {"width": 1440, "height": 900}},
})
try:
    with sync_playwright() as playwright:
        browser = playwright.chromium.connect_over_cdp(session["cdpBaseUrl"])
        context = browser.contexts[0]
        page = context.pages[0] if context.pages else context.new_page()
        page.goto("https://www.adscrawl.net")
        print(page.title())
        browser.close()
finally:
    client.cdp.close(session["sessionId"])

cdp.list() returns {"ok": true, "data": [...]}. cdp.get_version(session) discovers the WebSocket endpoint without forwarding the API key. Connection URLs contain secrets and should not be logged.

Persistent cloud browsers

import os
from adscrawl import AdsCrawl

client = AdsCrawl()
server = os.environ["ADSCRAWL_PROXY_SERVER"]
proxy = {"server": server}
launched = client.cloud_browsers.launch({
    "proxy": proxy,
    "tabs": ["https://www.adscrawl.net"],
})
browser_id = launched["id"]
try:
    profile = client.cloud_browsers.get(browser_id)
    print(profile["id"], profile["runtime"]["status"])
finally:
    client.cloud_browsers.stop(browser_id)

API-key starts and launches require a top-level custom proxy every time. A stopping response does not confirm shutdown; poll get() until stopped. Failed launches can expose a cleanup id as AdsCrawlAPIError.id.

Async client

import asyncio
from adscrawl import AsyncAdsCrawl

async def main():
    async with AsyncAdsCrawl() as client:
        markdown = await client.markdown({"url": "https://www.adscrawl.net"})
        print(markdown)

asyncio.run(main())

The async facade runs the dependency-free standard-library HTTP transport in worker threads. Cancelling a coroutine or reaching a timeout does not prove remote browser work stopped.

Configuration and errors

from adscrawl import AdsCrawl, AdsCrawlAPIError, AdsCrawlTimeoutError

client = AdsCrawl(base_url="https://api.adscrawl.net", timeout_ms=90_000)
try:
    text = client.markdown(
        {"url": "https://www.adscrawl.net", "timeoutMs": 60_000},
        timeout_ms=75_000,
    )
    print(text)
except AdsCrawlAPIError as error:
    print(error.status, error.code, error.trace_id)
except AdsCrawlTimeoutError:
    print("Inspect remote sessions before retrying.")

The API key defaults to ADSCRAWL_API_KEY; the base URL defaults to ADSCRAWL_BASE_URL, ADSCRAWL_API_URL, then https://api.adscrawl.net. Errors include AdsCrawlAPIError, AdsCrawlTimeoutError, AdsCrawlConnectionError, and AdsCrawlResponseError. API response bodies are credential-redacted. Requests are not automatically retried.

Develop and release

python -m unittest discover -s tests -v
python -m build
python -m twine check dist/*

Tests use fake responses and never consume AdsCrawl credits. See RELEASING.md for PyPI trusted publishing.

License

MIT