Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.3.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6909dae85a470ed12de2988529952a4f1d90062eec43f4c4b9970c07674d09f |
|
MD5 | 31905c3b0e9d17b85d37133b4a725fa3 |
|
BLAKE2b-256 | 767b567f58f5a6b46f5f77362f6e9fbd8fef9f4b27f4033525246c21a6325670 |
Close
Hashes for selenium_crawler_template-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6efc060a75d404e815bd493a24b737ac3164ceb0603f1f8f619a499b24405c5b |
|
MD5 | 95dd5a6f692778ce907037976059b0fb |
|
BLAKE2b-256 | 66f89a4460cefbce00d25906b8ead843e28b6462b1b9e3f777fbd5032ff11061 |