😁 Welcome!!

Contents

• 😁 Welcome!!
• Contents
• 🌐 Dynamic Web Scraper
• 🚀 Quick start
• 💡 Use case examples
• ✨ Features
• 📦 Installation
• 📲 Usage
• ⚙ Options
• ✏ Manual Installation
• ❌ Common errors
• 👥 Contributing

🌐 Dynamic Web Scraper

💻 Windows and Linux compatible. 💻

This is a dynamic web scraper specifically designed for websites that have to wait for certain elements to load (such as AngularJS). It runs at specified time intervals. This can be used to monitor when a new element is added to the website instead of having to manually refresh it.

Instead of looking at the source, it waits until all elements are loaded to retrieve the results by using selenium and a Firefox driver.

Whenever a new element is discovered, it will notify you and save it to a file so that it doesn't notify you again for that same element in the future.

🚀 Quick start

pipx install dynamic-scraper
dynamic-scraper -u "https://www.example.com" -s "search this text"

💡 Use case examples

This is useful, for example to notify you when a certain keyword is found on a website, such as:

• New job on a job board
• New product on an online store
• New article on a blog post
• ...

✨ Features

• Automated Scraping: Runs at user-defined intervals, extracting data without manual input.
• Notification System: Notifies users via Windows notifications when new data is found.
• Robust Parsing: Utilizes customizable search strings and regular expressions for data extraction.

📦 Installation

(Go below for manual installation.)

From PyPI

pipx is optional but recommended, you can use pip instead.

pipx:

pipx install dynamic-scraper

pip:

pip install dynamic-scraper

You can also clone the repository and install:

git clone https://github.com/P-ict0/Dynamic-Web-Scraper.git
cd Dynamic-Web-Scraper
pipx install .

📲 Usage

For help:

dynamic-scraper --help

General usage:

dynamic-scraper -u "https://www.example.com" -s "search this text"

Also see common errors if you encounter any issues with the browser.

⚙ Options

✏ Manual Installation

git clone https://github.com/P-ict0/AngularJS-Dynamic-Web-Scraper.git

Recommended to use a virtual environment:

python3 -m venv venv

source venv/bin/activate  # Linux
venv\Scripts\activate  # Windows

pip install -r requirements.txt

You can now run:

python src/web_scraper/scraper.py [args]

❌ Common errors

You may also need to install the latest geckodriver from here and add it to your PATH.

👥 Contributing

Contributions are welcome! Please fork the repository and submit a pull request with your suggested changes.