Skip to main content

Data App Crawler core

dataapp-crawler (imported as dataapp_crawler) is the provider-neutral website acquisition library for Python 3.11+. It can run independently of the DataApp Runtime. Core includes generic CDP/Playwright providers and an optional Camoufox adapter, but no Relay- or AWS-specific provider.

Dependency versions are defined in each package's pyproject.toml; an application uses its own uv.lock. Local source changes are not delivered to applications until matching packages and browser images are released and the application's lock is updated.

Basic acquisition

import asyncio

from dataapp_crawler import Crawler


async def main() -> None:
    async with Crawler() as crawler:
        capture = await crawler.fetch("https://example.com")
        print(capture.status)


asyncio.run(main())

Use an ordinary synchronous or asynchronous process(capture, context) function for extraction, OCR/ASR calls, model extraction, or custom logic. The SDK does not own a processing engine or Dataset:

async def process(capture, context):
    heading = capture.page.locator("h1").first
    await context.run(heading.wait_for(state="visible"), timeout=10)
    return {"title": (await context.run(heading.inner_text(), timeout=5)).strip()}

An operation timeout raises TimeoutError; exhausting Limits.max_duration raises LimitReached. Diagnostics are bounded and omit raw credentials, headers, connection URLs, response bodies, and arbitrary upstream exception text.

Page-driven flows

Crawler.run_browser(collect) hands one native Playwright Page and a CrawlContext to ordinary application code. A single collection function can submit a search, traverse list pages, open detail pages, and stop on business conditions. It does not require one URL per Crawler object.

from dataapp_crawler import Crawler, Limits
from dataapp_crawler.browser_tools import open_url, text_of


async def collect(page, context):
    await open_url(page, "https://example.com", context)
    return {"title": await text_of(page.locator("h1"), context, required=True)}


async with Crawler(browser=browser_provider, limits=Limits(max_duration=120)) as crawler:
    record = await crawler.run_browser(collect)

browser_tools includes bounded navigation and extraction helpers such as open_url, new_page, popup_page, same_tab, wait_until, text_of, attribute_of, and iterate_list. Page and Locator remain native Playwright objects. Raw Playwright actions remain available, but they do not automatically gain Crawler accounting, scheduling, diagnostics, or retry semantics.

The application defines selectors, stable business keys, pagination, readiness, and side-effect boundaries. Do not automatically retry clicks or submissions that can create duplicate actions.

Failure summaries include a fixed exception_type without its message. Browser flow failures also report root_host, the root Page's hostname only. This may be the list page rather than a failing detail tab. The original exception still propagates, with cancellation and cleanup unchanged. A failure before the root Page exists reports phase=page_creation.

To inspect a failing detail page, explicitly capture a bounded screenshot inside that page's scope before it closes. Keep artifacts private, outside uploaded source packages and logs. diagnostics_dir applies to request-driven captures; it does not automatically snapshot every native Page action in run_browser. An exception type or screenshot does not by itself establish whether the target site blocked access, returned a different layout, or had not finished loading.

Read-only CAPTCHA observations

detect_captchas(html, ...) and await detect_captchas_on_page(page, ...) classify observable reCAPTCHA v2/Enterprise, Turnstile, Cloudflare full-page Challenge, BotDetect, and image-CAPTCHA integrations. Results contain fixed evidence codes, component matches, and missing inputs. Script-only or unsupported evidence remains unknown; a form widget is distinct from an access challenge. CaptchaOverride accepts explicit site knowledge.

Detection does not navigate, click, capture images, submit forms, or contact paid services. A solver match is structural readiness, not authorization or proof of website acceptance. capture_image(locator) separately captures the currently rendered challenge for an explicitly chosen resource call. Keep submission and site acceptance checks in ordinary application code.

The candidate CloudflareChallengeCapture API observes dynamic full-page render parameters when installed before navigation, and applies a managed token only to the same active challenge. See the token workflow. The local contract and browser fixture have passed; real-site managed solving and deployment acceptance remain pending. Ordinary Turnstile and full-page Challenge tasks have different required fields. Detection alone never enables a paid call, and the cf_clearance cookie mode is not exposed.

See the source template's API reference for parameters and the Runtime/resource boundary.

Provider extensions

Supplier-specific provider implementations are separate distributions:

Distribution Import Public provider types
dataapp-crawler-octoparse-browser dataapp_crawler_octoparse BrowserEnvRelay, RelayBrowser
dataapp-crawler-agentcore dataapp_crawler_agentcore AgentCoreBrowser, AWSBrowser

The core wheel retains CDPBrowser, PlaywrightBrowser, and optional CamoufoxBrowser as generic integrations. It does not contain either supplier implementation and does not depend on Relay or boto3. Applications install one supplier extension deliberately and import its provider from that extension. This makes an accidental supplier-provider import fail at project-install time instead of silently exposing an unsupported runtime path.

Source packages may share this repository; that does not merge their wheel contents, dependency graphs, project templates, or base images.

Proxy boundary

Browser and proxy selection are independent. Core provides the common proxy contract and supports:

  • direct access with proxy=None;
  • a developer-provided StaticProxy;
  • an official proxy through DataOSProxy and the Resource Proxy client;
  • a custom ProxyProvider.acquire() returning a ProxyHandle.

DataOSProxy.proxy_group accepts auto, domestic_basic, overseas_basic, or overseas_enhanced. auto uses the basic official proxy service. The SDK does not select or name proxy suppliers: Resource Proxy owns the published supplier order and validates location and duration support.

If official proxy acquisition fails, DataOSProxy raises ConfigurationError with bounded code and status_code attributes. These diagnostics contain only the documented resource error class and HTTP status; supplier responses, proxy URLs, and credentials are never retained in the exception.

A proxy handle is tied to a session lifecycle. It is not a promise to rotate the exit IP on every request. Credentials must remain outside inputs, source files, logs, and build arguments.

DataApp integration

Within a Data App, use DataAppCrawler from bazhuayu_dataapp.plugins.crawler:

from bazhuayu_dataapp import DataApp
from bazhuayu_dataapp.plugins.crawler import DataAppCrawler


async with DataApp:
    async with DataAppCrawler(browser=browser_provider) as crawler:
        records = await crawler.run_browser(collect)
    await DataApp.push_data(records)

Runtime owns Run input, Dataset writes, progress, cancellation, and final state. DataAppCrawler only joins Crawler cancellation and cleanup to that lifecycle.

Logging

The standalone Crawler writes INFO stage logs to stderr by default. The DataApp adapter uses DataApp.log, so the platform receives the same bounded stages. Pass a standard logger to change verbosity:

import logging

logger = logging.getLogger("my_app.crawler")
logger.setLevel(logging.WARNING)

async with Crawler(logger=logger) as crawler:
    capture = await crawler.fetch(target_url)

The SDK does not configure the root logger and never logs the value returned by a process callback. Acquisition success is not Dataset success; the application must push records explicitly and validate the Runtime outcome separately.

Metadata

Release files for dataapp_crawler 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataapp_crawler 0.1.1
File Size Uploaded Provenance Actions
dataapp_crawler-0.1.1.tar.gz 78.7 kB Without provenance Details for dataapp_crawler-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dataapp_crawler 0.1.1
File Interpreter ABI Platform Actions
dataapp_crawler-0.1.1-py3-none-any.whl Python 3 none any Details for dataapp_crawler-0.1.1-py3-none-any.whl

Total release size: 174.4 kB

Release files / dataapp_crawler-0.1.1.tar.gz

Download URL dataapp_crawler-0.1.1.tar.gz
Size 78.7 kB
Tags Source
SHA-256 checksum
How to use checksums
c8d18a2ee143cb73b60bf9cd44fbcfa7feec7c6ee318257752123ac6f163dc70
BLAKE2b-256 checksum
How to use checksums
6192cab471ca7563a7a0f8f3d29651247a3e1650dd0727555e20081d10a2dc21
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / dataapp_crawler-0.1.1-py3-none-any.whl

Download URL dataapp_crawler-0.1.1-py3-none-any.whl
Size 95.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b95530ea25b516a77669ad9b651e1145a3a0db52d9dd3e9791f4e101cbb1e31c
BLAKE2b-256 checksum
How to use checksums
f50b1c9bc871d0100182dc55da55852e38a23348f9ff9794a27ccb719ca5a99e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for dataapp_crawler 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page