parsehub-dsl 1.0.1

A domain-specific language for web scraping, inspired by ParseHub

ParseHub DSL

ParseHub DSL is a bounded browser-automation DSL for structured web extraction.

Its officially supported runtime contract is:

• Playwright
• Chrome / Chromium-family browsers

The project is built around:

• a ParseHub-inspired interaction model
• a dedicated DSL grammar and compiler pipeline
• a Playwright-backed browser execution runtime
• a verified, bounded set of supported DSL examples and automation flows

The repo is no longer in broad modernization mode. Its latest bounded release mainline was:

release-to-1.0 convergence

That lane has now converged locally: the repo metadata and release-facing docs are aligned on a bounded 1.0.1 stable-release posture for the already supported modern DSL surface.

Current posture

• Phase 1: complete
• Phase 2: closeout documented; the former config-string runtime boundary is now closed by proof
• Phase 3: capability promotion review is complete
• Phase 4: docs / workflow / release-surface convergence is complete
• Post-Phase-4 bounded follow-ups: semantic authority, runtime authority, browser-confidence hardening, compiler/runtime hardening, and release-readiness slices are complete
• Latest bounded operator-surface expansion: local profile/session management now includes the closed RFC-anchored Phase 3 successor lane (config.session_profile default-profile selection + shared cross-profile session query seam; metadata indexing remains deferred behind the scale gate)
• Latest closed release lane: release-to-1.0-convergence
• Current repo metadata: parsehub-dsl==1.0.1 / Production/Stable
• Latest externally published artifact: parsehub-dsl==1.0.0 (the next repo-side patch target is 1.0.1)
• Historical prior publication milestone: parsehub-dsl==0.2.0b1
• Publication rule: repo verification now targets 1.0.1, while external stable publication remains a separate operator action; the latest published artifact stays 1.0.0 until that upload completes

Runtime support contract

ParseHub DSL is not positioned as a general-purpose “automate anything on any website” platform.

The current official support contract is narrower:

• supported runtime: Playwright
• supported browser family: Chrome / Chromium-family
• supported automation surface: the bounded DSL surface verified by this repository
• supported browser-confidence proof lane: deterministic local headless browser verification

Important note:

• the repository may retain internal browser abstractions for testing and architecture
• those internal seams do not imply public support for Selenium, Firefox, WebKit, or general multi-backend parity
• the current verified browser-confidence lane installs and launches Chromium via Playwright
• stealth-related code paths and flags remain optional and best-effort; they do not widen the support contract or create anti-bot / CAPTCHA / platform-defense success guarantees

Runtime operator controls

The supported operator surface is still bounded, but it includes practical runtime controls:

• --locale, --timezone-id, and --user-agent for browser-context shaping
• --browser-channel and --browser-executable for local Chromium-family launches
• --cdp-url to attach to an existing Chromium CDP endpoint instead of launching a local browser
• --disable-resources, --block-domain, and --extra-header for bounded request-layer control
• --disable-stealth for turning off optional best-effort stealth tweaks
• --profile, --session, and --profile-root for local profile/session-backed runs
• parsehub profile ... / parsehub session ... for local profile/session lifecycle management

Important operator notes:

• --browser-channel and --browser-executable are launch-only controls; they do not apply when --cdp-url is used
• --headed is a local debugging toggle for a local Playwright launch; it does not widen the supported proof story beyond the deterministic local headless browser-confidence lane
• these controls improve operator ergonomics and reproducibility, but they do not create public-site reliability, anti-bot, or multi-browser parity guarantees

Stable-release posture

Use these files as the authority chain:

• docs/START_HERE.md
• docs/README.md
• docs/CURRENT_STATUS.md
• docs/ROADMAP.md
• docs/RELEASE_CONTRACT.md
• docs/SUPPORT_POLICY.md
• CHANGELOG.md
• docs/EXPRESSION_MODEL.md
• docs/CAPABILITY_MATRIX.md
• docs/OMX_WORKING_CONVENTION.md
• docs/PROFILE_SESSION_ENVIRONMENT_MANUAL.md as the primary user-manual + design-appendix package
• docs/PROFILE_SESSION_ENVIRONMENT_GUIDE.md as a secondary companion for the full conceptual and CLI relationship
• docs/PROFILE_SESSION_ENVIRONMENT_QUICKSTART.md as a secondary companion for the shortest operator path
• docs/PROFILE_SESSION_ENVIRONMENT_DOMAIN_MODEL.md as a secondary companion for the designer-facing model
• docs/encoding.md (when diagnosing DSL/config/source-file encoding issues)

Additional operator references:

• docs/SUPPORTED_EXAMPLES.md
• docs/PARAMETER_INPUTS.md
• docs/TEST_LANES.md
• examples/USAGE_GUIDE.md

Install

Recommended: editable install

cd F:/parsehub/parsehub-dsl

python -m venv .venv
.venv\Scripts\activate

pip install -e ".[dev]"
python -m playwright install chromium

Alternative: requirements + editable package

cd F:/parsehub/parsehub-dsl

python -m venv .venv
.venv\Scripts\activate

pip install -r requirements.txt
python -m playwright install chromium
pip install -e .

Fast operator path

Installed CLI help

parsehub --help

Smallest supported execution path

parsehub example_simple.ph

Persist the full structured result

parsehub example_simple.ph --output json --output-file temp/example_simple.json

Create a local profile and capture a reusable login session

parsehub profile create demo_profile --name "Demo Profile" --login-url https://example.com/login
parsehub session login demo_profile
parsehub session verify sess_... --profile demo_profile
parsehub session export sess_... --profile demo_profile --output-dir temp/session-export
parsehub authenticated.ph --profile demo_profile

Use docs/PROFILE_SESSION_ENVIRONMENT_MANUAL.md as the primary authority for this topic. docs/PROFILES_AND_SESSIONS.md remains the specialized companion for storage/lifecycle/verification/export semantics, while the guide, quickstart, and domain-model docs remain secondary companions for narrower reading paths.

Manual login capture also supports bounded overrides such as --locale, --timezone-id, --user-agent, --browser-channel, --browser-executable, --disable-resources, repeated --extra-header, repeated --block-domain, and --disable-stealth.

Inspect parameter metadata without running the browser

parsehub examples/parameter_demo_current.ph --params-help
python -m parsehub.cli examples/parameter_demo_current.ph --params-help

Module entry is the development fallback after editable install (or with PYTHONPATH=src set). The installed parsehub console script is the canonical release-facing operator path.

Output behavior

The CLI now has two distinct output surfaces:

1. stdout preview
   • human-readable execution summary
   • grouped preview of collected rows
   • suitable for quick inspection
2. --output-file persisted result
   • full structured JSON or CSV output
   • suitable for downstream processing or saved artifacts

Important rule:

• --output controls the format of --output-file
• stdout remains a human-readable preview rather than raw JSON/CSV

DSL source-file encoding contract

• ParseHub DSL source files (.ph) are read as UTF-8
• files saved as UTF-8 with BOM are also accepted
• UTF-8 remains the only supported source-file encoding contract
• non-UTF-8 .ph files produce an explicit save the .ph file as UTF-8 guidance message
• non-UTF-8 --config JSON files produce an explicit save the .json file as UTF-8 guidance message
• non-UTF-8 internal grammar/resource files are reported as an internal install/project-file integrity problem rather than user DSL input drift

Default verification commands

Stable smoke baseline

pytest

Broader regression bundle

python run_phase1_regressions.py

Required release gates

python scripts/release/run_release_gates.py

Fresh-venv install / CLI / compile / run happy path

python scripts/release/run_release_happy_path.py

Publish-surface verification before manual release

python scripts/release/run_publish_surface_check.py

Explicit browser-confidence lane

pytest tests/runtime/test_playwright_driver.py tests/runtime/test_e2e_execution.py -m browser_confidence -q

Interpretation:

• pytest is the stable default baseline
• python run_phase1_regressions.py is the broader verified safety net
• python scripts/release/run_release_gates.py is the repo-owned release gate runner
• the browser-confidence lane remains non-default for plain pytest
• the same browser-confidence lane is now also invoked explicitly by release-gate automation
• the current explicit proof lane is Playwright-backed and uses locally installed Chromium
• python scripts/release/run_publish_surface_check.py validates the stable-release package artifacts and metadata locally
• none of these commands imply public-site scraping reliability

Supported example quickstart

Representative supported examples include:

• simple.ph
• product_list.ph
• nested_data.ph
• pipe_functions_demo.ph
• else_branch_demo.ph
• simple_param_test.ph
• enhanced_click_demo.ph
• label_system_demo.ph
• wait_time_units_demo.ph
• enhanced_click_demo_current.ph
• label_system_demo_current.ph
• wait_time_units_demo_current.ph
• parameter_demo_current.ph
• template_data_flow_current.ph
• search_interaction_scroll_current.ph
• runtime_engine_authority_current.ph

Useful compile commands:

python compile.py examples/parameter_demo_current.ph --verbose
python compile.py examples/template_data_flow_current.ph --verbose

See docs/SUPPORTED_EXAMPLES.md for the verified example-by-example guidance.

Known boundaries

The current release target is intentionally bounded.

Still out of scope:

• public-site scraping reliability guarantees
• headed/manual browser execution as a default guarantee
• anti-bot / CAPTCHA / stealth success guarantees
• stealth-related code paths, flags, or tests as a contract-widening signal
• official multi-browser runtime parity across Chrome / Firefox / WebKit
• official Selenium runtime support
• YAML config support
• legacy inline globals such as global counter = 0
• legacy dollar-ref syntax such as $counter
• legacy parameter aliases such as number / boolean

Docs

• start here / current project reality
• docs index
• current status snapshot
• roadmap / active next-step authority
• release contract / stable-release target
• support policy / runtime support contract
• changelog / release-facing history
• expression model / runtime contract
• capability matrix
• OMX working convention / repo-root workflow
• test lanes / verification surfaces
• supported examples quickstart
• parameter inputs and precedence
• profiles and sessions
• examples usage guide
• root-level legacy tests cleanup note
• legacy syntax migration guide

Repository layout

parsehub-dsl/
├── src/parsehub/     # grammar, parser, semantic analysis, IR, runtime, codegen
├── tests/            # smoke / integration / runtime verification lanes
├── examples/         # supported and historical DSL examples
├── docs/             # repo-visible authority docs
├── scripts/          # release, verification, and maintenance scripts
├── .github/          # CI / workflow configuration
├── compile.py        # compile / natural-language generation entrypoint
├── run_phase1_regressions.py
└── README.md

Practical next-step rule

When in doubt:

1. trust executable evidence first
2. trust top-level current-facts docs second
3. preserve the bounded release target
4. start any widened work from a new bounded plan instead of silently reopening old lanes