TQ-scroll-scrape is a library for downloading web pages via Selenium.
Project description
Scroll-Scraper
Python library for automating scrolling and downloading web pages via Selenium. This is especially useful for pages that utilize infinite scrolling.
Usage
Download ChromeDriver
Download ChromeDriver from https://chromedriver.chromium.org/downloads. Choose the version that matches the Chrome browser running on your system. Place the executable in the root of your Python project.
Build tqdnld Wheel
Fork or clone this repository and run python setup.py bdist_wheel
in the repository root.
Install tqdnld Wheel
In your consuming project, run pip install <path to wheel file>
.
Use the Module
Here is sample code demonstrating how to download a page:
from teqniqly.scroll_and_scrape import ScrollAndScrape
url = "https://www.espn.com/"
scroll_scraper = ScrollAndScrape()
scroll_scraper.download(url)
scroll_scraper.driver.close()
scroll_scraper.driver.quit()
This library will scroll the page until it reaches the end. The scroll height can be controlled
by passing the scroll_by
kwarg. If omitted the value of document.body.scrollHeight
is used.
To give items the chance to render, the library will wait before scrolling again. The wait time can
be controlled via the sleep_after_scroll_seconds
parameter. If omitted, the default value is two seconds.
The example below scrolls the page by 500 pixels, waiting 5 seconds between each scroll.
from teqniqly.scroll_and_scrape import ScrollAndScrape
url = "https://www.espn.com/"
downloader = ScrollAndScrape()
downloader.download(url, sleep_after_scroll_seconds=5, scroll_by=500)
downloader.driver.close()
downloader.driver.quit()
Post-download Processing
The download
function accepts a callback that executes after the page is downloaded. The page
source is passed into the callback. The example below saves the downloaded page to an html file.
from teqniqly.scroll_and_scrape import ScrollAndScrape
def save_file(source: str):
with open("espn.html", "w") as file:
file.write(source)
url = "https://www.espn.com/"
downloader = ScrollAndScrape()
downloader.download(url, save_file)
downloader.driver.close()
downloader.driver.quit()
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for tq_scroll_scrape-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7e7374e16484d2d2cfc778f6f72393fb28e76ba0afd16aec31926a810a43cc7 |
|
MD5 | 9f6a80f21c3c598b935d993447daacef |
|
BLAKE2b-256 | a120967b51321ce2cb67b164f59d6abf71ed5e19c0c9736b354b32adace1802b |