PlaywrightURLJsonExtractor

Two independent, deterministic CLI tools — no LLM/AI, no crawling:

render-url — renders one URL in headless Chromium (via Playwright) and outputs the post-JS DOM as JSON.
parse-html — parses the html field from render-url's output into structured JSON (via BeautifulSoup4).

URL -> render-url -> rendered JSON (html field) -> parse-html -> structured JSON

Install

pip install -e .
playwright install chromium

This puts render-url and parse-html on your PATH.

Commands

`render-url` — render a URL to JSON

render-url "https://example.com"

Prints one JSON object to stdout and writes it to an auto-incremented file (rendered_page_1.json, rendered_page_2.json, ...). Never overwrites existing files. With --raw-html, it prints and writes the rendered HTML instead (see Raw HTML mode).

Parameter	Flag	Default	Description
URL	positional, or `url` in config	(none)	The single URL to render. Required (from CLI or config).
Config file	`--config <path>`	`config.json`	Path to the JSON config file.
Timeout	`--timeout <ms>`	`30000`	Overall navigation timeout in milliseconds.
Stabilization	`--stabilization <ms>`	`1000`	Fixed settle time (ms) after load, before capturing the DOM.
Wait condition	`--wait-until <state>`	`load`	`load`, `domcontentloaded`, or `networkidle`.
Headless	`--headless` / `--no-headless`	`--headless`	Run Chromium headless or with a visible window.
Output prefix	`--output-prefix <name>`	`rendered_page`	Base name for the output JSON file.
Output dir	`--output-dir <dir>`	`.`	Directory the output file is written into.
Verbose	`--verbose` / `-v`	off	Log progress (navigation, waits, extraction, browser lifecycle) to stderr. Stdout still carries only the final JSON.
Log JSON	`--log-json`	off	Additionally pretty-print the final result JSON to stderr.
Detect	`--detect` / `--no-detect`	off	Check what kind of page this actually is before returning HTML. `--no-detect` overrides `"detect": true` in the config file. See Detecting what kind of page you got below.
Raw HTML	`--raw-html`	off	Print the rendered page's `<body>` markup (pretty-printed, no scripts/styles; not JSON) to stdout and write it to `<prefix>_N.html`. All other options still apply. If no HTML was produced (an error, or `--detect` found a non-application page), the JSON result is printed instead so you can see why.

Raw HTML mode (`--raw-html`)

Want the page's HTML itself rather than a JSON wrapper around it? Add --raw-html. You get clean, readable <body> markup: no scripts, styles or <head>, formatted one tag per line.

render-url "https://example.com" --raw-html                       # HTML on stdout + rendered_page_1.html
render-url "https://example.com" --raw-html > page.html           # capture just the HTML
render-url "https://example.com" --raw-html --stabilization 3000  # every other option still applies
render-url "https://example.com/job/1" --raw-html --no-detect     # skip page classification (needed if config has detect on)

Sample command — render a JS-heavy job page (skipping classification, since a plain listing has no application form), give it extra settle time, and save the HTML under a custom name and folder:

render-url "https://positrace.bamboohr.com/careers/122" --raw-html --no-detect --stabilization 3000 --output-dir out --output-prefix positrace

This prints the HTML to your terminal and writes it to out/positrace_1.html (the folder must already exist; the next run writes positrace_2.html). It always does both — there's no option to print without saving.

stdout is the post-JavaScript <body> markup only and nothing else: no <head>, and every <script>, <style>, <link>, <noscript> and <template> element is removed, along with inline style="..." and on*="..." (event handler) attributes. Only HTML remains, and it is pretty-printed (one tag per line, indented). (This means page metadata such as <title>, meta tags and JSON-LD (<script type="application/ld+json">) job data is not included; use the normal JSON mode if you need the full document.) Diagnostics (-v) still go to stderr.
The file written is <prefix>_N.html (auto-incremented, never overwrites) instead of .json.
Fallback: if there is no HTML to return — the render failed, or --detect classified the page as anything other than APPLICATION — the normal JSON result is printed instead (and saved as .json), so you can see what went wrong. Check whether stdout starts with {"ok": to tell the two apart.
Can also be set with "raw_html": true in config.json. If your config has "detect": true, non-application pages (e.g. a plain job listing, classified NO_FORM) hit the fallback above and return JSON — pass --no-detect to get the HTML anyway.

`parse-html` — extract structured data from rendered HTML

parse-html rendered_page_1.json

Reads a render-url JSON file (or stdin via -), extracts fields from its html, prints one JSON object to stdout, and writes it to an auto-incremented file (parsed_page_1.json, parsed_page_2.json, ...).

Parameter	Flag	Default	Description
Input file	positional, or `--input <path>`	(none)	`render-url` JSON file to read. Use `-` for stdin.
Config file	`--config <path>`	`config.json`	Path to the JSON config file (reads its `"parser"` section).
CSS selector	`--selector <css>`	`null`	Scopes `headings`/`links`/`images`/`text` to the first matching element. No match -> those fields come back empty, not an error.
Whitespace	`--no-strip-whitespace`	strip on	Disable whitespace collapsing in extracted text.
Output prefix	`--output-prefix <name>`	`parsed_page`	Base name for the output JSON file.
Output dir	`--output-dir <dir>`	`.`	Directory the output file is written into.
Verbose	`--verbose` / `-v`	off	Log progress (input loading, extraction steps, field counts) to stderr. Stdout still carries only the final JSON.
Log JSON	`--log-json`	off	Additionally pretty-print the final result JSON to stderr.

Extracted fields

Field	What it is	Fallback
`title`	`<title>` text	`null` if absent/empty
`headings`	`<h1>`–`<h6>`, in order	`[]` if none
`links`	`<a href>` text + href, in order (duplicates kept)	anchors without `href` excluded
`images`	`<img src>` + `alt`, in order	`alt` defaults to `""`; no `src` excluded
`meta`	`<meta name/property>` -> `content`	later tag wins on duplicate keys
`text`	visible body text (scripts/styles excluded)	`""` if none

Answering application questions with an LLM (optional)

parse-html and render-and-parse can optionally use an LLM (Anthropic Claude by default) to identify the application/screening questions on the page and answer them from your resume. This is the only LLM/network-calling feature in the project — everything else stays deterministic and offline; it's off by default and lives entirely behind --answer-questions.

--answer-questions requires --detect, and only runs when the page is classified APPLICATION. This keeps the LLM from ever being called on a CAPTCHA, login wall, closed job, bot challenge, or other non-application page. render-and-parse rejects --answer-questions without --detect immediately, before even launching Chromium. For the two-stage pipeline, parse-html --answer-questions reads the same signal from its input file's "status" field (present only when that file came from render-url --detect) — with no "status" field, or a "status" other than APPLICATION, it skips the LLM call and returns qa_error explaining why instead.

pip install -e ".[qa]"   # installs the anthropic SDK
cp .env.example .env     # fill in ANTHROPIC_API_KEY

Write your resume as Markdown in resume.md (gitignored — never commit it), then:

render-url "https://example.com/job/123" --detect
parse-html rendered_page_1.json --answer-questions

# or in one step:
render-and-parse "https://example.com/job/123" --detect --answer-questions

Parameter	Flag	Env var	Default	Description
Enable	`--answer-questions`	—	off	Turns the feature on. Also settable via `"answer_questions": true` in the `"parser"` section of `config.json`. Requires `--detect` (see above).
Resume	`--resume <path>`	—	`resume.md`	Path to your resume in Markdown; the LLM's source of truth when answering.
Provider	`--qa-provider <name>`	`LLM_PROVIDER`	`anthropic`	Only `anthropic` is currently implemented.
Model	`--qa-model <name>`	`ANTHROPIC_MODEL`	`sonnet5`	Short name (`sonnet5`, `opus5`, `haiku4.5`, ...) or a full model ID.
API key	`--qa-api-key <key>`	`ANTHROPIC_API_KEY`	—	Read from `.env` if not passed/set. Required unless the feature is off.

On success, two fields are added to the result's data:

{
  "application_questions": ["Are you legally authorized to work in Canada?"],
  "answers": [
    {"question": "Are you legally authorized to work in Canada?", "answer": "Yes."}
  ]
}

If the resume can't answer a question confidently, "answer" is null rather than a guess. If QA is blocked (missing --detect, page not classified APPLICATION) or fails for any other reason (missing resume, missing API key, LLM error), the deterministic parse result is still returned with a top-level "qa_error" string describing what happened — a QA failure or gate never discards the parse. render-and-parse with --answer-questions but no --detect is the one case that's a hard error instead ("error": {"type": "invalid_input", ...}), since it can be caught before any rendering happens.

`render-and-parse` — do both in one command

render-and-parse "https://example.com"

Runs render-url then parse-html in a single process and prints the final structured JSON. Writes both rendered_page_N.json and parsed_page_N.json.

Parameter	Flag	Default	Description
URL	positional	(none)	The URL to render and parse.
Config file	`--config <path>`	`config.json`	Shared config file for both stages.
Timeout	`--timeout <ms>`	`30000`	Render-stage navigation timeout.
Stabilization	`--stabilization <ms>`	`1000`	Render-stage settle time.
Wait condition	`--wait-until <state>`	`load`	Render-stage load state.
Headless	`--headless` / `--no-headless`	`--headless`	Render-stage Chromium visibility.
CSS selector	`--selector <css>`	`null`	Parse-stage extraction scope.
Whitespace	`--no-strip-whitespace`	strip on	Parse-stage whitespace handling.
Output dir	`--output-dir <dir>`	`.`	Directory for both output files.
Verbose	`--verbose` / `-v`	off	Log progress from both stages to stderr.
Log JSON	`--log-json`	off	Additionally pretty-print the render stage and final result JSON to stderr.
Detect	`--detect`	off	Check what kind of page this actually is before parsing it. See Detecting what kind of page you got below.

Detecting what kind of page you got

Some URLs — job application links especially — don't lead where you expect. You might land on a CAPTCHA, a login screen, a "this job is closed" page, an error page, or a bot-blocking screen like Cloudflare's "Checking your browser..." interstitial, instead of the actual page you wanted.

Passing --detect tells the tool to figure out which of these it landed on before handing you back the page content — using simple, predictable rules (checking for password fields, CAPTCHA widgets, known error/closed-job wording, and so on). No AI is involved; it's the same kind of exact, repeatable checking as everything else in this project.

render-url "https://example.com/job/123" --detect
render-and-parse "https://example.com/job/123" --detect

The result gets a status field telling you what was found:

`status`	Meaning
`APPLICATION`	Looks like a real application form. The page's HTML (or parsed data, for `render-and-parse`) is included as usual.
`CAPTCHA`	Blocked by a CAPTCHA.
`LOGIN`	You'd need to sign in first.
`ERROR`	An error page (404, 500, "something went wrong", etc).
`CLOSED_JOB`	The job posting says it's closed, filled, or expired.
`BOT_CHALLENGE`	A bot-protection screen (e.g. Cloudflare) that didn't clear after a short wait.
`NO_FORM`	The page loaded fine, but nothing that looks like an application form was found.

For anything other than APPLICATION, the HTML/parsed data comes back empty (null) — there was no point extracting it. You also get a detector object showing exactly which checks fired (captcha, login, bot_challenge, error, closed_job, form_detected), and a plain-text reason for the classification. This means downstream automation (or a human) can tell at a glance whether it's worth doing anything further with this URL, without having to read the page.

--detect is entirely optional — leave it off and both tools behave exactly as before.

Full pipeline manually (two commands)

render-url "https://example.com"
parse-html rendered_page_1.json

# or piped:
render-url "https://example.com" | parse-html -

Tests

python -m pytest tests/ -v

Publishing to PyPI

pip install build twine

# bump "version" in pyproject.toml first, then:
rm -rf dist build render_url.egg-info    
# PowerShell: Remove-Item -Recurse -Force dist, build, render_url.egg-info -ErrorAction SilentlyContinue
python -m build                # builds dist/*.whl and dist/*.tar.gz
python -m twine check dist/*   # validates metadata before upload
python -m twine upload dist/*  # uploads to PyPI (prompts for credentials/token)

Use python -m twine upload --repository testpypi dist/* to publish to TestPyPI first if you want to verify the package before a real release.

Configuration

Precedence: built-in defaults < config.json < CLI flags. Both tools share one config.json (parse-html only reads its "parser" key). Copy config.example.json to get started — config.json is gitignored (local/per-environment). Unknown keys are rejected as typos.

{
  "url": "https://example.com",
  "timeout_ms": 30000,
  "stabilization_ms": 1000,
  "wait_until": "load",
  "headless": true,
  "output_prefix": "rendered_page",
  "output_dir": ".",
  "verbose": false,
  "log_json": false,
  "detect": false,
  "parser": {
    "selector": null,
    "strip_whitespace": true,
    "output_prefix": "parsed_page",
    "output_dir": ".",
    "verbose": false,
    "log_json": false,
    "answer_questions": false,
    "resume_path": "resume.md",
    "qa_provider": "anthropic",
    "qa_model": "sonnet5"
  },
  "detector": {
    "threshold": 10,
    "challenge_wait_ms": 2000,
    "challenge_max_checks": 2
  }
}

The "detector" section only matters if you use --detect. challenge_wait_ms/challenge_max_checks control how long it waits and re-checks a page that looks like a bot-blocking screen before giving up. You can leave this section out entirely to use the sensible defaults shown above.

About threshold: every page gets a score built from evidence found in its form fields (an email input is worth 5 points, a resume upload 5, a "first name" field 2, and so on — see detector/constants.py for the full list). threshold is the minimum score a page needs to be called APPLICATION instead of NO_FORM.

Minimum useful value: 1. Anything this low means almost any form on the page (even a simple newsletter signup with just an email field) gets called an application — too loose to trust.
Maximum meaningful value: 45. That's the total score a page could ever get if it had every single positive signal the scorer looks for. Setting the threshold at or above this makes APPLICATION nearly impossible to reach, even for real, complete application forms — too strict to be useful.
Ideal / default: 10. In practice, a real job application form easily scores 20–30+ (email + text fields + textarea + file upload alone is already 20), while a stray contact form or newsletter box scores well under 10. 10 gives enough headroom to reject those false positives while still catching applications that are missing a field or two.

If you're seeing real application pages misclassified as NO_FORM, lower the threshold a bit (e.g. 7–8). If unrelated forms (contact forms, search bars, sign-up boxes) are being misclassified as APPLICATION, raise it (e.g. 12–15). Going far outside the 1–45 range defeats the point of having a threshold at all.

Output schemas

render-url success:

{"ok": true, "url": "...", "final_url": "...", "status_code": 200, "title": "...", "html": "...", "error": null}

Errors: invalid_url, navigation_error, navigation_timeout, browser_error, html_extraction_error, unknown_error, detection_error.

render-url with --detect adds these fields (see Detecting what kind of page you got):

{
  "ok": true, "url": "...", "final_url": "...", "status_code": 200, "title": "...",
  "html": "...",
  "status": "APPLICATION",
  "reason": "application_form_detected",
  "redirected": false,
  "detector": {"captcha": false, "login": false, "bot_challenge": false, "error": false, "closed_job": false, "form_detected": true},
  "form": {"count": 1, "inputs": 17, "textareas": 2, "selects": 3, "file_inputs": 1},
  "error": null
}

html is null whenever status isn't APPLICATION.

parse-html success:

{
  "ok": true,
  "source_url": "...",
  "title": "...",
  "data": {"headings": [...], "links": [...], "images": [...], "meta": {...}, "text": "..."},
  "selector_matched": null,
  "error": null
}

Errors: invalid_input, missing_html (Step 1 failed or had no html), invalid_config, parse_error, unknown_error.

When render-and-parse is run with --detect and the page isn't classified as APPLICATION, data comes back null and the same status/reason/detector/form/redirected fields as above are included instead.

parse-html with --answer-questions adds application_questions/answers to data on success (see Answering application questions with an LLM), or a top-level "qa_error" string if the --detect/APPLICATION gate blocked it or QA itself failed (the deterministic data above is still returned either way).

Both tools always print exactly one JSON object to stdout (success or failure) — diagnostics go to stderr only.

What this project intentionally does NOT do

No crawling, link-following, or multi-URL batching.
No DOM mutation, clicking, form submission, or authentication.
No LLM, AI, embeddings, or semantic interpretation in render-url/parse-html's core behavior — structural extraction and rule-based classification only. The sole exception is the opt-in --answer-questions feature (see above), which is off by default.
No solving or bypassing CAPTCHAs or bot-protection challenges — --detect only tells you one is there.
parse-html makes no network calls and never launches a browser.

Release files for render-url 1.1.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for render-url 1.1.4
File	Size	Uploaded
render_url-1.1.4.tar.gz	50.2 kB	Sep 18, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for render-url 1.1.4
File	Interpreter	ABI	Platform	Reset
render_url-1.1.4-py3-none-any.whl 36.3 kB Sep 18, 2026	Python 3	none	any	Details

Total release size: 86.4 kB

Release files / render_url-1.1.4.tar.gz

Download URL	render_url-1.1.4.tar.gz
Size	50.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`48059e177e01fce9f21b6f915c6239993b5872e5809594774738fee4ee410ee6`
BLAKE2b-256 checksum How to use checksums	`7d442e362b74ea6642c98bc5b56d0ae9a64ff3d685eabaefd914ec95beb06a8f`
Upload date	Sep 18, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.7`

Release files / render_url-1.1.4-py3-none-any.whl

Download URL	render_url-1.1.4-py3-none-any.whl
Size	36.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`3a242493fd61000c84bc8c04338f642ce5f722812f37145db42f2d11885c70c3`
BLAKE2b-256 checksum How to use checksums	`565a5f1ecc49b9d602383efab19dcbd7f1fbb3152cca6d42f8ddb6cffb8af38b`
Upload date	Sep 18, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.7`