Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8806b04a48485dc22c4698b9e1e9a46e71d86db64cd34e36fee13d0da75aec1 |
|
MD5 | 0663af913df198892dfbb90cb1924be8 |
|
BLAKE2b-256 | c8432efc0a03a4625ca061e6f30061b6b6aaf9100a667d3a631c3feed5822be3 |
Close
Hashes for selenium_crawler_template-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3cd39c78d331fbf1cc59ad07acfa736fb9fcc2b375b866786dadc27418b64c18 |
|
MD5 | 7cf9ecd7364a4b47f059736be05331a8 |
|
BLAKE2b-256 | 1554f2c8c0368ae5420a3ed337827ba2f29dbcf2b115aa0eff21b0512b382ed9 |