Boilerplate for developing crawler with Selenium.
Project description
Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for selenium-crawler-template-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0c547381582144f015b9631f572ec243a9221acbcbb12b18288cf56a4f627bb |
|
MD5 | 262def1244800bdf0917d357d883b9c0 |
|
BLAKE2b-256 | b578a2abbeff2e8d1f91583f2635a6a81bfea23787882973b93d409b9f1af474 |
Close
Hashes for selenium_crawler_template-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ff24b3fa20d42fb8cbc511b778b99c4da065057f11b9eafcfc3d6e37dd76ddd |
|
MD5 | beb9a410d91de4d0e2fc0694e32333e8 |
|
BLAKE2b-256 | 90955a942d93ba97a677848c107b0a20a5cd05d42c0fa7bedbb323f9f6dfb7c4 |