Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc9754c80b0adb00c62110ccf8918b2633f5f4c8d0468556b37388caacd43421 |
|
MD5 | 54533bd1221108749609de6fa7978d43 |
|
BLAKE2b-256 | 3770d1ae46b8cebef2d3e0f1a8889fb8a32af5e5c06e4484e044393d0dd5bca9 |
Close
Hashes for selenium_crawler_template-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa8ec51b678898979976865addd3aa39f1069c6e74b2c90dc2f0cd0306afe09a |
|
MD5 | c4f10daa3aef134e07bcb64a90429df8 |
|
BLAKE2b-256 | bd222247f57d6089ad88af7fa6ddb8416abdc7fa8bb8150fd86af7e1b2b015a4 |