WebScraper
- This program can scrap data from websites using different scrapers, and send an email when matches/ changes deadening on the scraper used
- There are 2 types of scrapers:
- Generic: Can scrap any website, but might not be as exact
- Specific: Can scrap only specific websites, but will be more exact
Generic Scrapers
- Text
- Diff
Specific Scrapers
- Cars.com
How to use
Text
- Set these specific env variables
-
SCRAPER=text # Scraper to use URL=<URL> # URL to scrape TEXT=<TEXT> # Text to look for
- Ensure all other required env variables are set
Diff
- Set these specific env variables
-
SCRAPER=diff # Scraper to use URL=<URL> # URL to scrape PERCENTAGE=<PERCENTAGE_DIFF> # Percentage difference to look for
- Ensure all other required env variables are set
Cars.com
- Set these specific env variables
-
SCRAPER=cars_com # Scraper to use URL=https://www.cars.com/shopping/results/ # URL to scrape, must be on the results page, for a specific search
- Ensure all other required env variables are set
Required env variables
SLEEP_TIME_SEC= # Time to sleep between each scrape
SENDER_EMAIL= # Email to send from
FROM_EMAIL= # Name to send from i.e. '"Web Scraper" <no-reply@jstockley.com>'
RECEIVER_EMAIL= # Email to send to
PASSWORD= # Password for the sender's email
SMTP_SERVER= # SMTP server to use
SMTP_PORT= # SMTP port to use
TLS= # True/False to use TLS
Running multiple of the same scraper
To run 2+ scrapers of the same type, i.e. 2 diff scrapers, make sure the host folder mapping is different
Ex:
diff-scraper-1:
image: jnstockley/web-scraper:latest
volumes:
- ./diff-scraper-1-data/:/app/data/
environment:
- TZ=America/Chicago
- SCRAPER=diff
- URL=https://google.com
- PERCENTAGE=5
- SLEEP_TIME_SEC=21600
diff-scraper-2:
image: jnstockley/web-scraper:latest
volumes:
- ./diff-scraper-2-data/:/app/data/
environment:
- TZ=America/Chicago
- SCRAPER=diff
- URL=https://yahoo.com
- PERCENTAGE=5
- SLEEP_TIME_SEC=21600
1 maintainer
Metadata
1 maintainer
Release files for js-web-scraper 1.2.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| js_web_scraper-1.2.21.tar.gz | 19.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| js_web_scraper-1.2.21-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 40.2 kB
Release files / js_web_scraper-1.2.21.tar.gz
| Download URL | js_web_scraper-1.2.21.tar.gz |
|---|---|
| Size | 19.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6a6515b3f7d82779aacceefeaa19e541d25ebf4a4867ffcf0cc5e22da4058758
|
|
BLAKE2b-256 checksum How to use checksums |
85ded092f5bf176eb3125c4ecb79c49e157076a8ec07c76b340fd71001ae84fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency logRelease files / js_web_scraper-1.2.21-py3-none-any.whl
| Download URL | js_web_scraper-1.2.21-py3-none-any.whl |
|---|---|
| Size | 21.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2634f27574340051ef8a799064ff6f82e476f64f476b291dccf5be6a94770cba
|
|
BLAKE2b-256 checksum How to use checksums |
354391806aab2c0bc4be782158a2e176a269c5e1fafac808c74846f8c36281d5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 3, 2026.
Transparency log
