Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a52fe8fc35fd9c599f3d0d498c143c4ffde6c6673ff3a4808c81a7aa5cb7775 |
|
MD5 | f91ea3d3ce40f2184ed5a5a1117c6ea8 |
|
BLAKE2b-256 | c07c7a8689f54300106fc21192df8913e484595d55a414c847cbad1b767ae58e |
Close
Hashes for selenium_crawler_template-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1de84f43aa0b846b68533d6170fa37d8f65f4e0f37a2870987f58cffdca87727 |
|
MD5 | 581ca03d3e444c46f3998d6f048f8a40 |
|
BLAKE2b-256 | 390c58489505105ef18385c12fd9aab637fec3ec97dc8a469f497c36e730690d |