Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd5129d14506bceaf42e5f19190275adafa9d0898a917f0a45df9ce4a414fc36 |
|
MD5 | 297fbf1e35125cf24d240910f75e9310 |
|
BLAKE2b-256 | 46574b91e2f8a1f572f9c6b06bff1bd2373f289864af4e9471c2f936b7fd7601 |
Close
Hashes for selenium_crawler_template-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04dde9691dbf7e873e06e242130e080c1f313008bc081b2f3e74d9cf660643e3 |
|
MD5 | 476cbdb1c005156c98a1a8487aa86b36 |
|
BLAKE2b-256 | aaa4a02894bcceb4854203bdd6706347a43721948a5a71b76a59c7c79225d912 |