Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | efee2df9b3e805d8b72ecdac58d7f2596c59e64eff0e3b33746d49aac5abc41a |
|
MD5 | e610f0ccba8924bff52f7c00036bcaa5 |
|
BLAKE2b-256 | fccc225dad8a4cac93c6bcf95546f25d92c8f277f0b649ba7e0dd469965a66f0 |
Close
Hashes for selenium_crawler_template-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03f83d3f9a82f72e3ea46c9ec31860d5f3b686515bd4c73ed1147fd5ae9f5582 |
|
MD5 | 636bfb6b027571f9c48e3ab3ad24e02b |
|
BLAKE2b-256 | d442796c8ae141a28951e68f2053de699fea05f8cf687eb8d83fdcc749311801 |