Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.3.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c94e72b85807e4efba12c3d52d50b2613e69960e7c89b4547d8c54db52c5f8e8 |
|
MD5 | bfe2f6e35a6419ad33b22df893908d2b |
|
BLAKE2b-256 | 72fd4d3678e93ab3710ad8e864383fc77038328d6b9a10f7088415c8093dab8b |
Close
Hashes for selenium_crawler_template-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2da6fc4a93e68c4f9e1be75ad145e3c832ccecfc80111f8af26acde57d05840 |
|
MD5 | 8533f480f7a8eb8bf68bd01695864919 |
|
BLAKE2b-256 | 9d49af25c712201bfc343b5e3c53315092c9d4df44dadd2453e825f643a9b1bf |