Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fa38dff9faf3f80695b1cdc7a6a22ebb46cc93e4cd296a8611d3f43e46b915c |
|
MD5 | a0215e3390600d51ea2127f530ec3175 |
|
BLAKE2b-256 | 84fcf31b40e5360842718107c7d5ff077a95f630dadcb90ade8e1c54748a8cce |
Close
Hashes for selenium_crawler_template-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c1b0f8254978e0baf8669dd8fa1e59448ca7a61a687216d171573f752fa0892 |
|
MD5 | 40f06529c15b48197112665493a7d414 |
|
BLAKE2b-256 | 9ac2ee8e901dbd9326679eee591591b5368f42df1a503819a4b772bde933f920 |