Skip to main content

A web scraping library for selenium and beautifulsoup

Project description

Here is a sample code for a good scraper logic with some validation logic. It helps to build a good scraper with validation steps and repeat logic.

Installation

pip install pyselenscrapr

Usage

from pyselenscraper.ScrapingBot import ScrapingBot, TakeScreenshotModes
from pyselenscraper.ScrapingLogic import ScrapingLogic
from pyselenscraper.ScrapingStep import ScrapingStep, ScrapingStepInterval, ScrapingStepErrorHandling

...
# Initialize selenium driver
driver = webdriver.Remote("http://localhost:4444/wd/hub", options=ff_options)
...

robot = ScrapingBot(driver, take_screenshots_mode=TakeScreenshotModes.Always)

try:

   search_command = "wikipedia matrix movies"

   robot.add_step(ScrapingStep("Check if there is a visible 'Accept all' or 'decline all' button",
                               lambda l: l.get_best_element("//button//div[contains(text(), 'All')]").click(),
                               interval=ScrapingStepInterval.BeforeAnyStep,
                               error_handling=ScrapingStepErrorHandling.Ignore,
                               can_execute=lambda l: l.current_url and "google" in l.current_url and l.is_visible("//button//div[contains(text(), 'All')]"),
                               was_executed=lambda l: not l.is_visible("//button//div[contains(text(), 'All')]"),),
                  step_group="default")

   ...
   # Your scraper Logic here
   ...

   finished_success = robot.run()

   if finished_success:
       print("The robot finished successfully")
   else:
       print("The robot did not finish successfully")

except Exception as e:
   log.error(e)
finally:
   driver.quit()

Documentation

The

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyselenscrapr-0.0.1.tar.gz (14.4 kB view hashes)

Uploaded Source

Built Distribution

pyselenscrapr-0.0.1-py3-none-any.whl (12.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page