Data App Crawler core
dataapp-crawler (imported as dataapp_crawler) is the provider-neutral website
acquisition library for Python 3.11+. It can run independently of the DataApp
Runtime. Core includes generic CDP/Playwright providers and an optional Camoufox
adapter, but no Relay- or AWS-specific provider.
Dependency versions are defined in each package's
pyproject.toml; an application uses its ownuv.lock. Local source changes are not delivered to applications until matching packages and browser images are released and the application's lock is updated.
Basic acquisition
import asyncio
from dataapp_crawler import Crawler
async def main() -> None:
async with Crawler() as crawler:
capture = await crawler.fetch("https://example.com")
print(capture.status)
asyncio.run(main())
Use an ordinary synchronous or asynchronous process(capture, context) function
for extraction, OCR/ASR calls, model extraction, or custom logic. The SDK does
not own a processing engine or Dataset:
async def process(capture, context):
heading = capture.page.locator("h1").first
await context.run(heading.wait_for(state="visible"), timeout=10)
return {"title": (await context.run(heading.inner_text(), timeout=5)).strip()}
An operation timeout raises TimeoutError; exhausting Limits.max_duration
raises LimitReached. Diagnostics are bounded and omit raw credentials, headers,
connection URLs, response bodies, and arbitrary upstream exception text.
Page-driven flows
Crawler.run_browser(collect) hands one native Playwright Page and a
CrawlContext to ordinary application code. A single collection function can
submit a search, traverse list pages, open detail pages, and stop on business
conditions. It does not require one URL per Crawler object.
from dataapp_crawler import Crawler, Limits
from dataapp_crawler.browser_tools import open_url, text_of
async def collect(page, context):
await open_url(page, "https://example.com", context)
return {"title": await text_of(page.locator("h1"), context, required=True)}
async with Crawler(browser=browser_provider, limits=Limits(max_duration=120)) as crawler:
record = await crawler.run_browser(collect)
browser_tools includes bounded navigation and extraction helpers such as
open_url, new_page, popup_page, same_tab, wait_until, text_of,
attribute_of, and iterate_list. Page and Locator remain native Playwright
objects. Raw Playwright actions remain available, but they do not automatically
gain Crawler accounting, scheduling, diagnostics, or retry semantics.
The application defines selectors, stable business keys, pagination, readiness, and side-effect boundaries. Do not automatically retry clicks or submissions that can create duplicate actions.
Failure summaries include a fixed exception_type without its message. Browser
flow failures also report root_host, the root Page's hostname only. This may be
the list page rather than a failing detail tab. The original exception still
propagates, with cancellation and cleanup unchanged. A failure before the root
Page exists reports phase=page_creation.
To inspect a failing detail page, explicitly capture a bounded screenshot inside
that page's scope before it closes. Keep artifacts private, outside uploaded
source packages and logs. diagnostics_dir applies to request-driven captures;
it does not automatically snapshot every native Page action in run_browser.
An exception type or screenshot does not by itself establish whether the target
site blocked access, returned a different layout, or had not finished loading.
Read-only CAPTCHA observations
detect_captchas(html, ...) and await detect_captchas_on_page(page, ...) classify
observable reCAPTCHA v2/Enterprise, Turnstile, Cloudflare full-page Challenge,
BotDetect, and image-CAPTCHA
integrations. Results contain fixed evidence codes, component matches, and missing
inputs. Script-only or unsupported evidence remains unknown; a form widget is
distinct from an access challenge. CaptchaOverride accepts explicit site knowledge.
Detection does not navigate, click, capture images, submit forms, or contact paid
services. A solver match is structural readiness, not authorization or proof of
website acceptance. capture_image(locator) separately captures the currently
rendered challenge for an explicitly chosen resource call. Keep submission and
site acceptance checks in ordinary application code.
The candidate CloudflareChallengeCapture API observes dynamic full-page
render parameters when installed before navigation, and applies a managed
token only to the same active challenge. See the token workflow.
The local contract and browser fixture have passed; real-site managed solving
and deployment acceptance remain pending. Ordinary Turnstile and full-page
Challenge tasks have different required fields. Detection alone never enables
a paid call, and the cf_clearance cookie mode is not exposed.
See the source template's API reference for parameters and the Runtime/resource boundary.
Provider extensions
Supplier-specific provider implementations are separate distributions:
| Distribution | Import | Public provider types |
|---|---|---|
dataapp-crawler-octoparse-browser |
dataapp_crawler_octoparse |
BrowserEnvRelay, RelayBrowser |
dataapp-crawler-agentcore |
dataapp_crawler_agentcore |
AgentCoreBrowser, AWSBrowser |
The core wheel retains CDPBrowser, PlaywrightBrowser, and optional
CamoufoxBrowser as generic integrations. It does not contain either supplier
implementation and does not depend on Relay or boto3. Applications install one
supplier extension deliberately and import its provider from that extension.
This makes an accidental supplier-provider import fail at project-install time
instead of silently exposing an unsupported runtime path.
Source packages may share this repository; that does not merge their wheel contents, dependency graphs, project templates, or base images.
Proxy boundary
Browser and proxy selection are independent. Core provides the common proxy contract and supports:
- direct access with
proxy=None; - a developer-provided
StaticProxy; - a managed
DataOSProxythrough the Resource Proxy client; - a custom
ProxyProvider.acquire()returning aProxyHandle.
DataOSProxy.proxy_group accepts auto, domestic_basic,
overseas_basic, or overseas_enhanced. auto uses the basic managed
service. The SDK does not select or name proxy suppliers: Resource Proxy owns
the published supplier order and validates location and duration support.
If managed proxy acquisition fails, DataOSProxy raises ConfigurationError
with bounded code and status_code attributes. These diagnostics contain only
the documented resource error class and HTTP status; supplier responses, proxy
URLs, and credentials are never retained in the exception.
A proxy handle is tied to a session lifecycle. It is not a promise to rotate the exit IP on every request. Credentials must remain outside inputs, source files, logs, and build arguments.
DataApp integration
Within a Data App, use DataAppCrawler from
bazhuayu_dataapp.plugins.crawler:
from bazhuayu_dataapp import DataApp
from bazhuayu_dataapp.plugins.crawler import DataAppCrawler
async with DataApp:
async with DataAppCrawler(browser=browser_provider) as crawler:
records = await crawler.run_browser(collect)
await DataApp.push_data(records)
Runtime owns Run input, Dataset writes, progress, cancellation, and final state.
DataAppCrawler only joins Crawler cancellation and cleanup to that lifecycle.
Logging
The standalone Crawler writes INFO stage logs to stderr by default. The DataApp
adapter uses DataApp.log, so the platform receives the same bounded stages.
Pass a standard logger to change verbosity:
import logging
logger = logging.getLogger("my_app.crawler")
logger.setLevel(logging.WARNING)
async with Crawler(logger=logger) as crawler:
capture = await crawler.fetch(target_url)
The SDK does not configure the root logger and never logs the value returned by a process callback. Acquisition success is not Dataset success; the application must push records explicitly and validate the Runtime outcome separately.
Metadata
Release files for dataapp_crawler 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataapp_crawler-0.1.0.tar.gz | 78.1 kB | Without provenance | Details for dataapp_crawler-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataapp_crawler-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for dataapp_crawler-0.1.0-py3-none-any.whl |
Total release size: 173.4 kB
Release files / dataapp_crawler-0.1.0.tar.gz
| Download URL | dataapp_crawler-0.1.0.tar.gz |
|---|---|
| Size | 78.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
10b40bd2d890df7072cc04eb7456268a8c03b52751f5ed0159e55d01610640e7
|
|
BLAKE2b-256 checksum How to use checksums |
5071196552f73b9b412cfd5a603bfd81ed64e8c67e5961679103c3158ae9c588
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / dataapp_crawler-0.1.0-py3-none-any.whl
| Download URL | dataapp_crawler-0.1.0-py3-none-any.whl |
|---|---|
| Size | 95.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6f058dc8b7918b2a0bf8403683fb136cc2b9d28d215f5b14770e9163e1253a17
|
|
BLAKE2b-256 checksum How to use checksums |
fecac2256f6cfa5b1ba8745c41c0893ef7888b65d1d1cd30a402a3fb985487c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for dataapp_crawler 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.