Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.4.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a376a824e3d91f20eecb9910d04b642dca856894e8fb80582f3e77e6949dc66 |
|
MD5 | 5bc55fdf7f6b82b2119ac05a6afde0cf |
|
BLAKE2b-256 | b7153a0f3923184a814aadac12d14c9db2ebff0de5dab615efe6d362b165dbb1 |
Close
Hashes for selenium_crawler_template-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65df5f2027ca91e1d7208ad84bf43e81e3e1ee2b177b9cd72d89248b6b76034d |
|
MD5 | bc449b7480505f98185052de09036558 |
|
BLAKE2b-256 | f1a3e0f1d965d85449e8d080a491eb6a3666c7d7980068061e1979e7806cb05b |