WebScraper-Caller

Small CLI that reads a JSON file of objects, calls render-and-parse on each object's url, and records the resulting output file paths back into the JSON for later use. The only field it cares about is url — everything else is optional and passed through untouched.

Installation

pip install scrape-applied-links
playwright install chromium

This installs the scrape-applied-links command along with its render-url dependency (which provides render-and-parse).

Installing from source instead:

pip install -e .
playwright install chromium

Input format

The tool only requires a url field on each object — everything else is optional and simply passed through untouched. It accepts either of two shapes for the input file:

A flat list of objects, each needing nothing but a url:

[
  { "url": "https://example.com/careers/123" },
  { "url": "https://example.com/careers/456" }
]

An object mapping arbitrary labels (e.g. dates) to lists of objects — useful if you want to group entries, such as by the date they were scraped:

{
  "2026-09-16": [
    { "url": "https://example.com/careers/123" },
    { "url": "https://example.com/careers/456" }
  ]
}

Objects can carry any extra fields you like (company, title, description, etc.) — they're ignored by this tool and left as-is. If a company field is present it's shown in log output while processing, but it isn't required.

After an object is processed, this tool adds two fields to it:

Usage

scrape-applied-links [json_file] [options]

• json_file (optional, positional) — path to the input JSON file (either input shape described above). Defaults to links_applied_by_date.json in the current directory.

Options

Examples

Process the default file:

scrape-applied-links

Preview which URLs would be scraped, without calling anything:

scrape-applied-links --dry-run

Re-run and re-scrape every job, even ones already processed:

scrape-applied-links --force

Use a different input file and write render output to a subfolder:

scrape-applied-links my_links.json --output-dir out/

What it does

1. Loads the JSON file (a flat list or a label-keyed object of lists) and collects every object that has a url.
2. Skips objects that already have a rendered_page value, unless --force is passed.
3. For each remaining object, runs render-and-parse <url> (in --output-dir if given).
4. Detects the newly created rendered_page_N.json and parsed_page_N.json files and adds their paths to that job's entry as rendered_page and parsed_page.
5. Saves the JSON file after each successfully processed job, so progress isn't lost if a later URL fails.
6. If any URLs failed, prints a summary of them at the end and exits with a non-zero status.