Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0bbd14490f74abe37471b859cf28357e0282ac17c55cda25cd5e62eb00423c8 |
|
MD5 | be1f0729fcc655bffe5acfbaf1aac863 |
|
BLAKE2b-256 | 9b6538c38307ff8d587cd486a2d8fb3b39a992f9762cc5f8e6ede02cd51badff |
Close
Hashes for selenium_crawler_template-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1fe799ccf80fb49cead60e8c94a5d12289a8927414e2f58a3940eb296d30f30 |
|
MD5 | 6892fabe7a0006c40a5960b75afd283f |
|
BLAKE2b-256 | 5066e3447e2fcc12f667cf485747ecb4a5ed22069f74e8faae29d612b28ba6c9 |