aventure-scraper

Ethical, robots.txt-first CLI for Avature-hosted career sites. It is meant as a reference implementation: polite delays, incremental JSON cache, progressive fetch (HTTP → headless Chromium → headful Chromium), and explicit legal gates on disallowed URLs.

Features

Install

From the repo (editable):

cd avature_scraper_src
pip install -e ".[dev]"

For Playwright fallbacks:

pip install -e ".[browser]"
python -m playwright install chromium

Basic run

aventure-scraper https://company.avature.net/careers --output jobs.json

Pin explicit job detail URLs (repeatable):

aventure-scraper https://company.avature.net/careers \
  --job-url "https://company.avature.net/careers/JobDetail/Example-Role/12345" \
  --output jobs.json

Output JSON

Each saved job includes at least:

• id — Requisition-style id from the URL/body when possible; otherwise a stable url-<sha256-prefix> derived from the normalized listing URL.
• title, description (full cleaned visible text, capped for cache size; see description_truncated), location, url.
• needs_manual_review — Set when headful inference or ingest-date substitution was used; such rows are not dropped by the heuristic bogus filter on cache write.
• data_quality_warnings — Short operator-facing notes.

The report also includes output_fields: the column names implied by your CLI flags (defaults only, or the full JobDataPool-style list when using --all).

Optional column flags (JobDataPool-shaped)

These turn on extraction + completion checks for the matching columns (missing values on requests/headless cause escalation to the next tier, same as garbage-page detection):

--all (JobDataPool export mode)

--all is shorthand for enabling all of the flags above and expanding output_fields to the full JobDataPool-style column set defined in code (JOBDATAPOOL_OUTPUT_FIELDS in models.py), suitable for CSV export toward jobdatapool.com / JobPool pipelines.

aventure-scraper https://company.avature.net/careers --all --output jobs.json

CSV export and JobPool upload

Default behavior: if the run produced at least one job, the tool writes {output-stem}.csv beside your JSON report (UTF-8 BOM, JobPool-shaped headers) and POSTs the listing payload to https://jobpool.live/api/scrape-cache (override with JOBPOOL_SCRAPE_CACHE_URL). The POST body is JSON listing objects, not the raw CSV file.

The CSV uses headers compatible with the public scrape cache (REQUIRED_FIELDS in the livejobpool Netlify function), including source_observed_utc and record_lifecycle_state (left blank here; the API normalizes them).

Skip only the network upload (still write the default CSV when there are jobs):

aventure-scraper https://company.avature.net/careers --output jobs.json --no-upload-to-jobpool

Custom CSV path (upload still on unless you add --no-upload-to-jobpool):

aventure-scraper https://company.avature.net/careers --output jobs.json --csv-out ./out/listings.csv

Environment variables

Example (Windows PowerShell):

$env:LIVEJOBPOOL_ROOT = "C:\Users\you\Projects\livejobpool"
aventure-scraper https://company.avature.net/careers --output jobs.json

Who is user_name on the wire?

The cache API expects user_name for leaderboards. Resolution order:

1. --jobpool-user if you passed it
2. git config github.user
3. GitHub handle parsed from git config remote.origin.url (or upstream) when it looks like git@github.com:handle/repo.git or https://github.com/handle/repo
4. Slugified git config user.name
5. OS login (getpass.getuser())
6. Machine hostname

Output cache behavior (JSON)

If --output already exists, you get a small TTY selector: continue from cache or overwrite. The JSON is written incrementally after discovery and after each successful job fetch.

Raw landing HTML is not kept in the cache file on disk (only enough for discovery resume).

Legal gate

If robots.txt disallows a URL, the tool stops unless you type the exact acknowledgement phrase shown in the UI. This is deliberate friction; the tool does not provide legal advice.

Development

pip install -e ".[dev]"
pytest tests/ -q
ruff check src tests

Related projects

• jobpool.live — transparency layer and public scrape-cache API used above.
• jobdatapool.com — schema / interchange expectations for job rows.
• livejobpool — Netlify site + cache-scraped-listings function backing the cache.