basic-web-scraper 0.12.0a3

Basic Web Scraper made with selenium and bs4

Basic Web Scraper

Project Description

This package can be used for simple automated web surfing / scraping.

Additionally, the included BasicSpider class is meant to be extended by inheritance.

---

Usage Example

from basic_web_scraper.BasicSpider import BasicSpider

class CustomSpider(BasicSpider):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def custom_operation(self, threshold):
        """
        Scroll to predefined threshold.
        If past threshold, scroll back up.
        """
        if self.get_page_y_offset() < threshold:
            self.mousewheel_vscroll(number_of_scrolls=2)

        else:
            y_difference = self.get_page_y_offset() - threshold
            self.smooth_vscroll_up_by(y_difference)

---

Details about files

geckodriver.exe | geckodriver

Keep this file in local directory when using scraper on windows. If you're using a linux system, make sure to include geckodriver in PATH variable. Downloaded from here

Note: geckodriver.exe is for windows, and geckodriver (no extension) is for Linux

BasicSpider.py

Use this as the superclass for your own project's spider

This Spider can do basic things like goto a url, scroll down the page in different ways, refresh the page, etc..

It acts as an interface to selenium.webdriver to make setting up a project easier

custom_exceptions.py

Copy alongside BasicSpider.py, add any custom exceptions to this file.

package.json | package-lock.json

used to install dependencies related to testing and its related workflows. not needed for the spider to function.

start_local_server.py

used for testing. optional.

tests.py

this is where all the tests for the scraper are done. optional.

mock_webpage

used for testing. optional. can be used as a simple demo.