A web scraper that combines both Beautiful Soup (bs4) and Selenium.
Project description
A web scraper that combines both Beautiful Soup (bs4) and Selenium.
Want to support the development and stay updated?
Installation
pip install PyWebScraper
Usage
from PyWebScraper import Scraper
Scraper() will load bs4 or selenium (defined with the input 'scraper_type') to then load a website (defined by the input 'url') and save it under Scraper().page.
Optional inputs for Scraper():
url = str (will be opened in scraper and page saved in Scraper().page)
scraper_type = 'bs4' or 'selenium'
scroll_down = boolean (scrolls down in selenium first before saving page)
user_agent = 'desktop' or 'mobile'
auto_close_selenium = boolean (if False, you can further interact with the selenium browser via Scraper().selenium)
selenium_remote_webdriver = str (IP for a remote webdriver for selenium, see https://www.selenium.dev/docs/site/en/remote_webdriver/remote_webdriver_client/)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
PyWebScraper-0.1.5.tar.gz
(4.9 kB
view hashes)
Built Distribution
Close
Hashes for PyWebScraper-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41a147d365d9803a27d7c1e37eea3c713ab2d1a6f5a4e557054ab69f7b68a2fe |
|
MD5 | add5d7af4029ac6d41e441d360eccbb6 |
|
BLAKE2b-256 | 2849d2b68f4d8047ab83b8afe067a4d1d89ad6d327f07beff797c326d45ba87a |