Skip to main content

WebScraper

  • This program can scrap data from websites using different scrapers, and send an email when matches/ changes deadening on the scraper used
  • There are 2 types of scrapers:
    • Generic: Can scrap any website, but might not be as exact
    • Specific: Can scrap only specific websites, but will be more exact

Generic Scrapers

  • Text
  • Diff

Specific Scrapers

  • Cars.com

How to use

Text

  1. Set these specific env variables
  2.  SCRAPER=text # Scraper to use
     URL=<URL> # URL to scrape
     TEXT=<TEXT> # Text to look for
    
  3. Ensure all other required env variables are set

Diff

  1. Set these specific env variables
  2.  SCRAPER=diff # Scraper to use
     URL=<URL> # URL to scrape
     PERCENTAGE=<PERCENTAGE_DIFF> # Percentage difference to look for
    
  3. Ensure all other required env variables are set

Cars.com

  1. Set these specific env variables
  2.  SCRAPER=cars_com # Scraper to use
     URL=https://www.cars.com/shopping/results/ # URL to scrape, must be on the results page, for a specific search
    
  3. Ensure all other required env variables are set

Required env variables

SLEEP_TIME_SEC= # Time to sleep between each scrape
SENDER_EMAIL= # Email to send from
FROM_EMAIL= # Name to send from i.e. '"Web Scraper" <no-reply@jstockley.com>'
RECEIVER_EMAIL= # Email to send to
PASSWORD= # Password for the sender's email
SMTP_SERVER= # SMTP server to use
SMTP_PORT= # SMTP port to use
TLS= # True/False to use TLS

Running multiple of the same scraper

To run 2+ scrapers of the same type, i.e. 2 diff scrapers, make sure the host folder mapping is different Ex:

  diff-scraper-1:
    image: jnstockley/web-scraper:latest
    volumes:
      - ./diff-scraper-1-data/:/app/data/
    environment:
      - TZ=America/Chicago
      - SCRAPER=diff
      - URL=https://google.com
      - PERCENTAGE=5
      - SLEEP_TIME_SEC=21600

  diff-scraper-2:
    image: jnstockley/web-scraper:latest
    volumes:
      - ./diff-scraper-2-data/:/app/data/
    environment:
      - TZ=America/Chicago
      - SCRAPER=diff
      - URL=https://yahoo.com
      - PERCENTAGE=5
      - SLEEP_TIME_SEC=21600

Metadata

Release files for js-web-scraper 1.2.27

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for js-web-scraper 1.2.27
File Size Uploaded Provenance Actions
js_web_scraper-1.2.27.tar.gz 19.2 kB With provenance Details for js_web_scraper-1.2.27.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for js-web-scraper 1.2.27
File Interpreter ABI Platform Actions
js_web_scraper-1.2.27-py3-none-any.whl Python 3 none any Details for js_web_scraper-1.2.27-py3-none-any.whl

Total release size: 40.4 kB

Release files / js_web_scraper-1.2.27.tar.gz

Download URL js_web_scraper-1.2.27.tar.gz
Size 19.2 kB
Tags Source
SHA-256 checksum
How to use checksums
a8d41f78d89fe87845854296d41938a96e6974e00edca5882b9020eef0697639
BLAKE2b-256 checksum
How to use checksums
1fc237e3d7796138c8aee4888d69fec3047303781d68e4ea8cf4a4d640bb759f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / js_web_scraper-1.2.27-py3-none-any.whl

Download URL js_web_scraper-1.2.27-py3-none-any.whl
Size 21.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
67f15fbcc71e8d7627bc4d8f31223529762d6ace604a8530152cb6c651b3af6f
BLAKE2b-256 checksum
How to use checksums
7cd514cd8f3771f096b96d60f32bc694c1a5e37faa269a07633b0ae72e25fb1d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for js-web-scraper 1.2.27

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for js_web_scraper-1.2.27.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.27 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page