PlaywrightURLJsonExtractor

Two independent, deterministic CLI tools — no LLM/AI, no crawling:

• render-url — renders one URL in headless Chromium (via Playwright) and outputs the post-JS DOM as JSON.
• parse-html — parses the html field from render-url's output into structured JSON (via BeautifulSoup4).

URL -> render-url -> rendered JSON (html field) -> parse-html -> structured JSON

Install

pip install -e .
playwright install chromium

This puts render-url and parse-html on your PATH.

Commands

render-url — render a URL to JSON

render-url "https://example.com"

Prints one JSON object to stdout and writes it to an auto-incremented file (rendered_page_1.json, rendered_page_2.json, ...). Never overwrites existing files.

parse-html — extract structured data from rendered HTML

parse-html rendered_page_1.json

Reads a render-url JSON file (or stdin via -), extracts fields from its html, prints one JSON object to stdout, and writes it to an auto-incremented file (parsed_page_1.json, parsed_page_2.json, ...).

Extracted fields

render-and-parse — do both in one command

render-and-parse "https://example.com"

Runs render-url then parse-html in a single process and prints the final structured JSON. Writes both rendered_page_N.json and parsed_page_N.json.

Full pipeline manually (two commands)

render-url "https://example.com"
parse-html rendered_page_1.json

# or piped:
render-url "https://example.com" | parse-html -

Tests

python -m pytest tests/ -v

Publishing to PyPI

pip install build twine

# bump "version" in pyproject.toml first, then:
rm -rf dist build render_url.egg-info    # PowerShell: Remove-Item -Recurse -Force dist, build, render_url.egg-info -ErrorAction SilentlyContinue
python -m build                # builds dist/*.whl and dist/*.tar.gz
python -m twine check dist/*   # validates metadata before upload
python -m twine upload dist/*  # uploads to PyPI (prompts for credentials/token)

Use python -m twine upload --repository testpypi dist/* to publish to TestPyPI first if you want to verify the package before a real release.

Configuration

Precedence: built-in defaults < config.json < CLI flags. Both tools share one config.json (parse-html only reads its "parser" key). Copy config.example.json to get started — config.json is gitignored (local/per-environment). Unknown keys are rejected as typos.

{
  "url": "https://example.com",
  "timeout_ms": 30000,
  "stabilization_ms": 1000,
  "wait_until": "load",
  "headless": true,
  "output_prefix": "rendered_page",
  "output_dir": ".",
  "verbose": false,
  "log_json": false,
  "parser": {
    "selector": null,
    "strip_whitespace": true,
    "output_prefix": "parsed_page",
    "output_dir": ".",
    "verbose": false,
    "log_json": false
  }
}

Output schemas

render-url success:

{"ok": true, "url": "...", "final_url": "...", "status_code": 200, "title": "...", "html": "...", "error": null}

Errors: invalid_url, navigation_error, navigation_timeout, browser_error, html_extraction_error, unknown_error.

parse-html success:

{
  "ok": true,
  "source_url": "...",
  "title": "...",
  "data": {"headings": [...], "links": [...], "images": [...], "meta": {...}, "text": "..."},
  "selector_matched": null,
  "error": null
}

Errors: invalid_input, missing_html (Step 1 failed or had no html), invalid_config, parse_error, unknown_error.

Both tools always print exactly one JSON object to stdout (success or failure) — diagnostics go to stderr only.

What this project intentionally does NOT do

• No crawling, link-following, or multi-URL batching.
• No DOM mutation, clicking, form submission, or authentication.
• No LLM, AI, embeddings, or semantic interpretation — structural extraction only.
• parse-html makes no network calls and never launches a browser.