Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15faec5de50166032c0144ec520d56f573ac7c452a8f80918c9ede763d2805fd |
|
MD5 | c14d4adaff5317e9600176e255f2a5f3 |
|
BLAKE2b-256 | 7e21efdd1e5b9e93f905e2bb373bf77900ef1e59fce35032af01577dc5c68350 |
Close
Hashes for selenium_crawler_template-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f91a673c07ad35d594c64b9bb95fc3f564d377dd9e9272066a88b778ec5f57f |
|
MD5 | b50ed2c93504b5e17e41ab579c81e6f2 |
|
BLAKE2b-256 | 35a0dd22c7bd07fcb9c24bf08c7b29fbc4004ce971bf006add699c2bdeda1199 |