Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85b4ecafcd5d6e40ef22c223bfde5a8f4d1e8e26acb7aa1bd767b627cdd98a95 |
|
MD5 | 4bcb878dbc92673ec7021413c77c51ce |
|
BLAKE2b-256 | 4fc51876645c3d9e685940561a27d6ce46f90486eb75f0a6192251b31fd8e1fc |
Close
Hashes for selenium_crawler_template-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d7c46859cb943754574519226bd80994186078c229fdf474d06592cd002220d |
|
MD5 | 950a4c00b78fb18728a9a361efa469b4 |
|
BLAKE2b-256 | 7e9a255b1486e4b701c8dc401ce0baae79018835ba070f98de72627c3068bb6a |