Octoparse Crawler SDK
Octoparse browser providers for DataApp Crawler.
The accompanying browser-image source is maintained in
browserenv/. Python provider packages and the BrowserEnv
image share this repository but are built and released separately; BrowserEnv is
excluded from the Python wheel and sdist.
Install the browser package from the DataOS development index. It installs
Crawler and the Relay Python transport as dependencies. The Python import path
is dataapp_crawler_octoparse.
Install
pip install --extra-index-url https://nexus.bazhuayu.com/repository/pypi-dataos-dev/simple/ \
dataapp-crawler-octoparse-browser==0.1.0.dev4
Providers
BrowserEnvRelayconnects to the Relay browser supplied by the DataApp Relay image.RelayBrowserstarts and owns a local browser, Relay extension, and Relay server.
from dataapp_crawler_octoparse import BrowserEnvRelay
browser = BrowserEnvRelay()
The SDK's BrowserEnv image bundles CloakBrowser and runs it in headed mode,
using Xvfb when needed. Applications connect through BrowserEnvRelay.
RelayBrowser is an advanced self-managed path, not the Relay Data App template
default. A fresh post-fix local candidate batch passed full Relay pairing,
HTTP/HTTPS DOM work, owned-process cleanup, and container cleanup with Chrome
151 in headless and Xvfb modes, CloakBrowser 146 headless, and Chromium 156
headless. Earlier Chrome pairing and CloakBrowser headless failures remain useful
pre-fix history, not the current result. These are versioned local candidate
checks, not universal browser or target-site guarantees; proxy combinations were
not repeated across every browser. The published clean rc.3 template image is
a separate release artifact from the locally tested
dataos-octoparse-integration:20261108-managed candidate.
BrowserEnv proxy control uses DATAAPP_BROWSER_PROXY_API_URL,
DATAAPP_BROWSER_PROXY_API_USERNAME, and DATAAPP_BROWSER_PROXY_API_PASSWORD.
It also accepts the legacy OCTOPUS_PROXY_SOFTWARE_API_* settings when their
canonical equivalents are absent. The control endpoint must be HTTP on
loopback. An upstream https:// proxy uses TLS with certificate and hostname
verification.
In a DataHub Run, choose DataOSProxy(DataApp.resources()) as the
DataAppCrawler proxy provider to attribute browser traffic to the Run.
Select a lease duration that covers the browser work. Renewal depends on the
configured supplier.
CAPTCHA and images
AWS WAF image selection and controlled extension grants are internal C10/C07 experiments. The current platform Runtime does not expose their routes. Direct local Resource Proxy experiments require matching Relay, extension and server candidates. Local protocol checks do not establish real-site submission, clearance, data collection or availability through a platform Run.
CAPTCHA solving is opt-in. Install the managed extra with the local candidate
Resource Proxy client. In local development, open ResourceProxy.from_env();
platform applications use DataApp.resources() when the corresponding
Runtime/DataHub routes are enabled. Model routes still require that platform
integration; local support does not imply availability on pre.
For local managed-resource testing, save the current environment's DataOS API Key in a regular text file containing exactly one non-empty line, then run:
python -m dataapp_crawler run --api-key-file /path/to/api-key.txt -- python app.py
The source file cannot be a symlink or directory. The local launcher reads it locally and injects it only into the child process. Resource Proxy receives the normal Authorization header, never the source file. Do not paste the Key into chat or package the source file with the App. Platform Runs use the injected Runtime identity.
The typed API accepts embedded Turnstile, reCAPTCHA v2, Enterprise v2, and
image-to-text tasks. Widget tasks return .token; image tasks return .text.
Module selection chooses a recognizer for a known image technology, rather
than detecting what a website uses:
from dataapp_crawler.images import capture_image
image = await context.run(capture_image(page.locator("#captcha-image")))
task = await context.run(resources.solve_captcha(
type="image_to_text",
image_body=image.image_body,
capmonster_module="botdetect", # Only for a known BotDetect challenge.
))
capture_image() defaults to a visible rendered-element capture and does not
request a mutable image URL again. Chromium uses a clipped CDP screenshot when
available and otherwise falls back to the native Locator screenshot; hidden or
zero-size elements fail safely. When the developer has established that a
loaded IMG is the current challenge, source="loaded_image" explicitly copies
its pixels through Canvas without another GET and remains subject to cross-origin
pixel restrictions. The same image's data_url can be passed to
resources.chat_completion() as an inline model image. These small helpers do
not detect, solve, submit, or verify a challenge, and they do not turn process
into a processing framework. The developer still checks the
challenge identity, applies the candidate, checks the site verdict, and extracts
business data. Use the matching version's capability acceptance matrix to check
which task and site flows have been validated.
In the final real Relay check, default capture rejected the hidden zero-size IMG,
while explicit loaded_image returned its current PNG bytes without another
request. A visible wrapper capture no longer timed out, but that page's wrapper
pixels were blank; a successful screenshot therefore does not prove it contains
the challenge image or that a solver or site will accept it.
The managed cloud API through DataApp.resources() and Resource Proxy is the
default public solving path. Application code must explicitly create a bounded
task after the developer has enabled metered solving; CAPTCHA detection alone
never creates a paid task. In a local Runtime/DataHub resource-bridge acceptance,
the managed cloud API
passed one public reCAPTCHA Enterprise v2 demonstration form through real token
creation, browser submission, server acceptance, and Dataset output. BotDetect
image and vision-model candidates were submitted but rejected by the
demonstration site. Turnstile and regular reCAPTCHA v2 did not receive the same
acceptance test. Detection only classifies evidence; it does not solve a
challenge. None of these results establishes success on a business site.
Native CapMonster extension with a developer-owned key (advanced opt-in)
For trusted local development, a developer may explicitly configure BrowserEnv's bundled native CapMonster extension before browser startup:
# CAPMONSTER_API_KEY is already loaded from the developer's local secret store.
export CAPMONSTER_API_KEY
export CAPMONSTER_CAPTCHA_TYPES='ReCaptcha2,ReCaptchaEnterprise,Turnstile'
xvfb-run -a octoparse-relay-task -- uv run python app.py
CAPMONSTER_CAPTCHA_TYPES is a comma-separated allowlist. Each selected type
must be supported by the extension bundle in the chosen image. The key belongs
to the developer and is not a shared platform key. Do not commit it, pass it as
a build argument, write it to logs or Dataset records, or expose it in errors.
The launcher prepares a private extension copy before browser startup and removes
the two variables from the child application environment; the key still enters
the third-party extension context.
This BYOK path is currently limited to trusted local sessions. The launcher rejects raw CapMonster keys when DataHub platform identity is present, so this configuration is not a supported production DataApp path. A Resource Proxy grant-backed automatic extension integration is under development and is not a public product API. Use the explicit managed cloud API where platform credential isolation is required. Do not run native automatic solving and explicit API solving against the same challenge. Native Enterprise end-to-end site acceptance has not yet been established.
Documentation and examples
The Relay project template ships the following together:
- Developer guide: browser lifecycles, extension input behavior, fingerprints, proxies, and limits.
- API reference: Provider, Crawler helper, and advanced low-level Relay signatures and return values.
- Importable scenarios: pointer observation, scoped navigation, readiness, and extraction.
- Complete search/list/detail flow:
the site's business flow used by the Runtime
app.pyentry.
Page and Locator remain native Playwright objects. Pointer trajectory handling
is implemented by the extension; it is not an additional Python human_click()
method. Browser fingerprint behavior belongs to the selected browser environment.
Metadata
Release files for dataapp-crawler-octoparse-browser 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dataapp_crawler_octoparse_browser-0.1.0.tar.gz | 17.8 kB | Without provenance | Details for dataapp_crawler_octoparse_browser-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl |
Total release size: 38.1 kB
Release files / dataapp_crawler_octoparse_browser-0.1.0.tar.gz
| Download URL | dataapp_crawler_octoparse_browser-0.1.0.tar.gz |
|---|---|
| Size | 17.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3b0c04b81cec754a6465357d45281ded07f4b048566ce074831353f134fae165
|
|
BLAKE2b-256 checksum How to use checksums |
c0d0f583a9d6c9b3e9ab643a55eb089dfb38124e242f60f61e54d7d44bb18560
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl
| Download URL | dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl |
|---|---|
| Size | 20.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
59f51fb776aa5772c12129f91931597605cdcadbf3d518f9da5cd169ff8ccb8f
|
|
BLAKE2b-256 checksum How to use checksums |
9846fd1c5be0e879cb78249b5a6a7bfc3f5d55634cc2c2ced321861c23adb6d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for dataapp-crawler-octoparse-browser 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.