pooled-job-scraper 0.1.2

Generic job listings scraper with baseline dedupe and integration with the open jobpool

Pooled Job Scraper

pooled-job-scraper is published on PyPI and provides a CLI for scraping business careers pages, extracting listings, and generating a unique delta against the April 2026 baseline dataset.

PyPI package:

• https://pypi.org/project/pooled-job-scraper/

Platform Links

• mewannajob.com
• jobpool.live

How They Fit Together

• jobpool.live is the open data pool and hydration surface.
  Use this scraper to discover and normalize job listings, then review unique delta rows before promoting data into the pool workflow.
• mewannajob.com is the consumer-facing experience for browsing and using listings data.
  Data prepared through the pool process ultimately supports downstream job discovery use cases there.

Documentation Surfaces

• Public pool context and hydration navigation: jobpool.live
• Hydration docs in this repository: pool/hydration/docs/
• Scraper implementation in this repository: scripts/generic_job_listings_scraper.py

Install

Windows

py -m pip install --upgrade pooled-job-scraper

WSL / Linux / macOS

python3 -m pip install --upgrade pooled-job-scraper

Usage

job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv

Default Cache Behavior

By default, unique rows are posted to:

• https://jobpool.live/api/scrape-cache

job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv

The scraper infers user_name from local environment or git config and sends:

• user_name
• request_timestamp
• source_business_urls
• listings (standard listing fields plus any additional discovered fields)

Disable Cache Submission

Use the only cache-related flag when you need to skip cache persistence:

job-scraper \
  --business-url https://mossyhonda.hireology.careers/ \
  --company-name "Mossy Honda" \
  --disable-cache \
  --output output/mossy-scraped.csv \
  --unique-output output/mossy-unique.csv

Cache API

• POST /api/scrape-cache stores a scrape request payload.
• GET /api/scrape-cache?limit=25&user_name=<name> returns recent cached submissions.

Publishing Flow

Publishing is automated through:

• .github/workflows/publish-pypi.yml

Behavior:

• Triggers on push/merge to main.
• Builds distributions from pyproject.toml.
• Checks whether the current version already exists on PyPI.
• Publishes only when the version is new.
• Skips cleanly when that version already exists.

To release a new version:

1. Bump project.version in pyproject.toml.
2. Merge to main.
3. Wait for the publish workflow to complete.