Skip to main content

Data App Crawler core

dataapp-crawler (imported as dataapp_crawler) is the provider-neutral website acquisition library for Python 3.11+. It can run independently of the DataApp Runtime. Core includes generic CDP/Playwright providers and an optional Camoufox adapter, but no Relay- or AWS-specific provider.

Dependency versions are defined in each package's pyproject.toml; an application uses its own uv.lock. Local source changes are not delivered to applications until matching packages and browser images are released and the application's lock is updated.

Basic acquisition

import asyncio

from dataapp_crawler import Crawler


async def main() -> None:
    async with Crawler() as crawler:
        capture = await crawler.fetch("https://example.com")
        print(capture.status)


asyncio.run(main())

Use an ordinary synchronous or asynchronous process(capture, context) function for extraction, OCR/ASR calls, model extraction, or custom logic. The SDK does not own a processing engine or Dataset:

async def process(capture, context):
    heading = capture.page.locator("h1").first
    await context.run(heading.wait_for(state="visible"), timeout=10)
    return {"title": (await context.run(heading.inner_text(), timeout=5)).strip()}

An operation timeout raises TimeoutError; exhausting Limits.max_duration raises LimitReached. Diagnostics are bounded and omit raw credentials, headers, connection URLs, response bodies, and arbitrary upstream exception text.

Page-driven flows

Crawler.run_browser(collect) hands one native Playwright Page and a CrawlContext to ordinary application code. A single collection function can submit a search, traverse list pages, open detail pages, and stop on business conditions. It does not require one URL per Crawler object.

from dataapp_crawler import Crawler, Limits
from dataapp_crawler.browser_tools import open_url, text_of


async def collect(page, context):
    await open_url(page, "https://example.com", context)
    return {"title": await text_of(page.locator("h1"), context, required=True)}


async with Crawler(browser=browser_provider, limits=Limits(max_duration=120)) as crawler:
    record = await crawler.run_browser(collect)

browser_tools includes bounded navigation and extraction helpers such as open_url, new_page, popup_page, same_tab, wait_until, text_of, attribute_of, and iterate_list. Page and Locator remain native Playwright objects. Raw Playwright actions remain available, but they do not automatically gain Crawler accounting, scheduling, diagnostics, or retry semantics.

The application defines selectors, stable business keys, pagination, readiness, and side-effect boundaries. Do not automatically retry clicks or submissions that can create duplicate actions.

Failure summaries include a fixed exception_type without its message. Browser flow failures also report root_host, the root Page's hostname only. This may be the list page rather than a failing detail tab. The original exception still propagates, with cancellation and cleanup unchanged. A failure before the root Page exists reports phase=page_creation.

To inspect a failing detail page, explicitly capture a bounded screenshot inside that page's scope before it closes. Keep artifacts private, outside uploaded source packages and logs. diagnostics_dir applies to request-driven captures; it does not automatically snapshot every native Page action in run_browser. An exception type or screenshot does not by itself establish whether the target site blocked access, returned a different layout, or had not finished loading.

Read-only CAPTCHA observations

detect_captchas(html, ...) and await detect_captchas_on_page(page, ...) classify observable reCAPTCHA v2/Enterprise, Turnstile, Cloudflare full-page Challenge, BotDetect, and image-CAPTCHA integrations. Results contain fixed evidence codes, component matches, and missing inputs. Script-only or unsupported evidence remains unknown; a form widget is distinct from an access challenge. CaptchaOverride accepts explicit site knowledge.

Detection does not navigate, click, capture images, submit forms, or contact paid services. A solver match is structural readiness, not authorization or proof of website acceptance. capture_image(locator) separately captures the currently rendered challenge for an explicitly chosen resource call. Keep submission and site acceptance checks in ordinary application code.

The candidate CloudflareChallengeCapture API observes dynamic full-page render parameters when installed before navigation, and applies a managed token only to the same active challenge. See the token workflow. The local contract and browser fixture have passed; real-site managed solving and deployment acceptance remain pending. Ordinary Turnstile and full-page Challenge tasks have different required fields. Detection alone never enables a paid call, and the cf_clearance cookie mode is not exposed.

See the source template's API reference for parameters and the Runtime/resource boundary.

Provider extensions

Supplier-specific provider implementations are separate distributions:

Distribution Import Public provider types
dataapp-crawler-octoparse-browser dataapp_crawler_octoparse BrowserEnvRelay, RelayBrowser
dataapp-crawler-agentcore dataapp_crawler_agentcore AgentCoreBrowser, AWSBrowser

The core wheel retains CDPBrowser, PlaywrightBrowser, and optional CamoufoxBrowser as generic integrations. It does not contain either supplier implementation and does not depend on Relay or boto3. Applications install one supplier extension deliberately and import its provider from that extension. This makes an accidental supplier-provider import fail at project-install time instead of silently exposing an unsupported runtime path.

Source packages may share this repository; that does not merge their wheel contents, dependency graphs, project templates, or base images.

Proxy boundary

Browser and proxy selection are independent. Core provides the common proxy contract and supports:

  • direct access with proxy=None;
  • a developer-provided StaticProxy;
  • a managed DataOSProxy through the Resource Proxy client;
  • a custom ProxyProvider.acquire() returning a ProxyHandle.

DataOSProxy.proxy_group accepts auto, domestic_basic, overseas_basic, or overseas_enhanced. auto uses the basic managed service. The SDK does not select or name proxy suppliers: Resource Proxy owns the published supplier order and validates location and duration support.

If managed proxy acquisition fails, DataOSProxy raises ConfigurationError with bounded code and status_code attributes. These diagnostics contain only the documented resource error class and HTTP status; supplier responses, proxy URLs, and credentials are never retained in the exception.

A proxy handle is tied to a session lifecycle. It is not a promise to rotate the exit IP on every request. Credentials must remain outside inputs, source files, logs, and build arguments.

DataApp integration

Within a Data App, use DataAppCrawler from bazhuayu_dataapp.plugins.crawler:

from bazhuayu_dataapp import DataApp
from bazhuayu_dataapp.plugins.crawler import DataAppCrawler


async with DataApp:
    async with DataAppCrawler(browser=browser_provider) as crawler:
        records = await crawler.run_browser(collect)
    await DataApp.push_data(records)

Runtime owns Run input, Dataset writes, progress, cancellation, and final state. DataAppCrawler only joins Crawler cancellation and cleanup to that lifecycle.

Logging

The standalone Crawler writes INFO stage logs to stderr by default. The DataApp adapter uses DataApp.log, so the platform receives the same bounded stages. Pass a standard logger to change verbosity:

import logging

logger = logging.getLogger("my_app.crawler")
logger.setLevel(logging.WARNING)

async with Crawler(logger=logger) as crawler:
    capture = await crawler.fetch(target_url)

The SDK does not configure the root logger and never logs the value returned by a process callback. Acquisition success is not Dataset success; the application must push records explicitly and validate the Runtime outcome separately.

Metadata

Release files for dataapp_crawler 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataapp_crawler 0.1.0
File Size Uploaded Provenance Actions
dataapp_crawler-0.1.0.tar.gz 78.1 kB Without provenance Details for dataapp_crawler-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dataapp_crawler 0.1.0
File Interpreter ABI Platform Actions
dataapp_crawler-0.1.0-py3-none-any.whl Python 3 none any Details for dataapp_crawler-0.1.0-py3-none-any.whl

Total release size: 173.4 kB

Release files / dataapp_crawler-0.1.0.tar.gz

Download URL dataapp_crawler-0.1.0.tar.gz
Size 78.1 kB
Tags Source
SHA-256 checksum
How to use checksums
10b40bd2d890df7072cc04eb7456268a8c03b52751f5ed0159e55d01610640e7
BLAKE2b-256 checksum
How to use checksums
5071196552f73b9b412cfd5a603bfd81ed64e8c67e5961679103c3158ae9c588
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / dataapp_crawler-0.1.0-py3-none-any.whl

Download URL dataapp_crawler-0.1.0-py3-none-any.whl
Size 95.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6f058dc8b7918b2a0bf8403683fb136cc2b9d28d215f5b14770e9163e1253a17
BLAKE2b-256 checksum
How to use checksums
fecac2256f6cfa5b1ba8745c41c0893ef7888b65d1d1cd30a402a3fb985487c5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for dataapp_crawler 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page