Skip to main content

Octoparse Crawler SDK

Octoparse browser providers for DataApp Crawler.

The accompanying browser-image source is maintained in browserenv/. Python provider packages and the BrowserEnv image share this repository but are built and released separately; BrowserEnv is excluded from the Python wheel and sdist.

Install the browser package from the DataOS development index. It installs Crawler and the Relay Python transport as dependencies. The Python import path is dataapp_crawler_octoparse.

Install

pip install --extra-index-url https://nexus.bazhuayu.com/repository/pypi-dataos-dev/simple/ \
  dataapp-crawler-octoparse-browser==0.1.0.dev4

Providers

  • BrowserEnvRelay connects to the Relay browser supplied by the DataApp Relay image.
  • RelayBrowser starts and owns a local browser, Relay extension, and Relay server.
from dataapp_crawler_octoparse import BrowserEnvRelay

browser = BrowserEnvRelay()

The SDK's BrowserEnv image bundles CloakBrowser and runs it in headed mode, using Xvfb when needed. Applications connect through BrowserEnvRelay.

RelayBrowser is an advanced self-managed path, not the Relay Data App template default. A fresh post-fix local candidate batch passed full Relay pairing, HTTP/HTTPS DOM work, owned-process cleanup, and container cleanup with Chrome 151 in headless and Xvfb modes, CloakBrowser 146 headless, and Chromium 156 headless. Earlier Chrome pairing and CloakBrowser headless failures remain useful pre-fix history, not the current result. These are versioned local candidate checks, not universal browser or target-site guarantees; proxy combinations were not repeated across every browser. The published clean rc.3 template image is a separate release artifact from the locally tested dataos-octoparse-integration:20261108-managed candidate.

BrowserEnv proxy control uses DATAAPP_BROWSER_PROXY_API_URL, DATAAPP_BROWSER_PROXY_API_USERNAME, and DATAAPP_BROWSER_PROXY_API_PASSWORD. It also accepts the legacy OCTOPUS_PROXY_SOFTWARE_API_* settings when their canonical equivalents are absent. The control endpoint must be HTTP on loopback. An upstream https:// proxy uses TLS with certificate and hostname verification.

In a DataHub Run, choose DataOSProxy(DataApp.resources()) as the DataAppCrawler proxy provider to attribute browser traffic to the Run. Select a lease duration that covers the browser work. Renewal depends on the configured supplier.

CAPTCHA and images

AWS WAF image selection and controlled extension grants are internal C10/C07 experiments. The current platform Runtime does not expose their routes. Direct local Resource Proxy experiments require matching Relay, extension and server candidates. Local protocol checks do not establish real-site submission, clearance, data collection or availability through a platform Run.

CAPTCHA solving is opt-in. Install the managed extra with the local candidate Resource Proxy client. In local development, open ResourceProxy.from_env(); platform applications use DataApp.resources() when the corresponding Runtime/DataHub routes are enabled. Model routes still require that platform integration; local support does not imply availability on pre.

For local managed-resource testing, save the current environment's DataOS API Key in a regular text file containing exactly one non-empty line, then run:

python -m dataapp_crawler run --api-key-file /path/to/api-key.txt -- python app.py

The source file cannot be a symlink or directory. The local launcher reads it locally and injects it only into the child process. Resource Proxy receives the normal Authorization header, never the source file. Do not paste the Key into chat or package the source file with the App. Platform Runs use the injected Runtime identity.

The typed API accepts embedded Turnstile, reCAPTCHA v2, Enterprise v2, and image-to-text tasks. Widget tasks return .token; image tasks return .text. Module selection chooses a recognizer for a known image technology, rather than detecting what a website uses:

from dataapp_crawler.images import capture_image

image = await context.run(capture_image(page.locator("#captcha-image")))
task = await context.run(resources.solve_captcha(
    type="image_to_text",
    image_body=image.image_body,
    capmonster_module="botdetect",  # Only for a known BotDetect challenge.
))

capture_image() defaults to a visible rendered-element capture and does not request a mutable image URL again. Chromium uses a clipped CDP screenshot when available and otherwise falls back to the native Locator screenshot; hidden or zero-size elements fail safely. When the developer has established that a loaded IMG is the current challenge, source="loaded_image" explicitly copies its pixels through Canvas without another GET and remains subject to cross-origin pixel restrictions. The same image's data_url can be passed to resources.chat_completion() as an inline model image. These small helpers do not detect, solve, submit, or verify a challenge, and they do not turn process into a processing framework. The developer still checks the challenge identity, applies the candidate, checks the site verdict, and extracts business data. Use the matching version's capability acceptance matrix to check which task and site flows have been validated.

In the final real Relay check, default capture rejected the hidden zero-size IMG, while explicit loaded_image returned its current PNG bytes without another request. A visible wrapper capture no longer timed out, but that page's wrapper pixels were blank; a successful screenshot therefore does not prove it contains the challenge image or that a solver or site will accept it.

The managed cloud API through DataApp.resources() and Resource Proxy is the default public solving path. Application code must explicitly create a bounded task after the developer has enabled metered solving; CAPTCHA detection alone never creates a paid task. In a local Runtime/DataHub resource-bridge acceptance, the managed cloud API passed one public reCAPTCHA Enterprise v2 demonstration form through real token creation, browser submission, server acceptance, and Dataset output. BotDetect image and vision-model candidates were submitted but rejected by the demonstration site. Turnstile and regular reCAPTCHA v2 did not receive the same acceptance test. Detection only classifies evidence; it does not solve a challenge. None of these results establishes success on a business site.

Native CapMonster extension with a developer-owned key (advanced opt-in)

For trusted local development, a developer may explicitly configure BrowserEnv's bundled native CapMonster extension before browser startup:

# CAPMONSTER_API_KEY is already loaded from the developer's local secret store.
export CAPMONSTER_API_KEY
export CAPMONSTER_CAPTCHA_TYPES='ReCaptcha2,ReCaptchaEnterprise,Turnstile'
xvfb-run -a octoparse-relay-task -- uv run python app.py

CAPMONSTER_CAPTCHA_TYPES is a comma-separated allowlist. Each selected type must be supported by the extension bundle in the chosen image. The key belongs to the developer and is not a shared platform key. Do not commit it, pass it as a build argument, write it to logs or Dataset records, or expose it in errors. The launcher prepares a private extension copy before browser startup and removes the two variables from the child application environment; the key still enters the third-party extension context.

This BYOK path is currently limited to trusted local sessions. The launcher rejects raw CapMonster keys when DataHub platform identity is present, so this configuration is not a supported production DataApp path. A Resource Proxy grant-backed automatic extension integration is under development and is not a public product API. Use the explicit managed cloud API where platform credential isolation is required. Do not run native automatic solving and explicit API solving against the same challenge. Native Enterprise end-to-end site acceptance has not yet been established.

Documentation and examples

The Relay project template ships the following together:

Page and Locator remain native Playwright objects. Pointer trajectory handling is implemented by the extension; it is not an additional Python human_click() method. Browser fingerprint behavior belongs to the selected browser environment.

Metadata

Release files for dataapp-crawler-octoparse-browser 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataapp-crawler-octoparse-browser 0.1.0
File Size Uploaded Provenance Actions
dataapp_crawler_octoparse_browser-0.1.0.tar.gz 17.8 kB Without provenance Details for dataapp_crawler_octoparse_browser-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dataapp-crawler-octoparse-browser 0.1.0
File Interpreter ABI Platform Actions
dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl Python 3 none any Details for dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl

Total release size: 38.1 kB

Release files / dataapp_crawler_octoparse_browser-0.1.0.tar.gz

Download URL dataapp_crawler_octoparse_browser-0.1.0.tar.gz
Size 17.8 kB
Tags Source
SHA-256 checksum
How to use checksums
3b0c04b81cec754a6465357d45281ded07f4b048566ce074831353f134fae165
BLAKE2b-256 checksum
How to use checksums
c0d0f583a9d6c9b3e9ab643a55eb089dfb38124e242f60f61e54d7d44bb18560
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl

Download URL dataapp_crawler_octoparse_browser-0.1.0-py3-none-any.whl
Size 20.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
59f51fb776aa5772c12129f91931597605cdcadbf3d518f9da5cd169ff8ccb8f
BLAKE2b-256 checksum
How to use checksums
9846fd1c5be0e879cb78249b5a6a7bfc3f5d55634cc2c2ced321861c23adb6d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for dataapp-crawler-octoparse-browser 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page