Selenium Crawler Template
Boilerplate for developing crawler with Selenium.
Installation
pip install selenium-crawler-template
Usage
from selenium_crawler_template import Crawler
class MyCrawler(Crawler):
@Crawler.open_url_in_new_tab
def _get_email_from_profile(self, _):
return self.find_element('a#email').get_attribute('href')
def crawl(self, **kwargs):
self.driver.get(kwargs['url'])
for profile in self.find_elements('ul > .profile'):
_ = self._get_email_from_profile(profile.get_attribute('href'))
self._scroll_to_bottom()
1 maintainer
Release files for selenium-crawler-template 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| selenium-crawler-template-0.4.0.tar.gz | 3.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| selenium_crawler_template-0.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.5 kB
Release files / selenium-crawler-template-0.4.0.tar.gz
| Download URL | selenium-crawler-template-0.4.0.tar.gz |
|---|---|
| Size | 3.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3a376a824e3d91f20eecb9910d04b642dca856894e8fb80582f3e77e6949dc66
|
|
BLAKE2b-256 checksum How to use checksums |
b7153a0f3923184a814aadac12d14c9db2ebff0de5dab615efe6d362b165dbb1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.8.5
|
Release files / selenium_crawler_template-0.4.0-py3-none-any.whl
| Download URL | selenium_crawler_template-0.4.0-py3-none-any.whl |
|---|---|
| Size | 3.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
65df5f2027ca91e1d7208ad84bf43e81e3e1ee2b177b9cd72d89248b6b76034d
|
|
BLAKE2b-256 checksum How to use checksums |
f1a3e0f1d965d85449e8d080a491eb6a3666c7d7980068061e1979e7806cb05b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.8.5
|
