Pyppeteer Components for Scrapy & Gerapy
Project description
Gerapy Pyppeteer
This is a package for supporting pyppeteer in Scrapy, also this package is a module in Gerapy.
Installation
pip3 install gerapy-pyppeteer
Usage
You can use PyppeteerRequest
to specify a request which uses pyppeteer to render.
For example:
yield PyppeteerRequest(detail_url, callback=self.parse_detail)
And you also need to enable PyppeteerMiddleware
in DOWNLOADER_MIDDLEWARES
:
DOWNLOADER_MIDDLEWARES = {
'gerapy_pyppeteer.downloadermiddlewares.py.PyppeteerMiddleware': 543,
}
Others optional settings:
# pyppeteer logging level
GERAPY_PYPPETEER_LOGGING_LEVEL = logging.WARNING
# pyppeteer timeout
GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT = 30
# pyppeteer browser window
GERAPY_PYPPETEER_WINDOW_WIDTH = 1400
GERAPY_PYPPETEER_WINDOW_HEIGHT = 700
# pyppeteer settings
GERAPY_PYPPETEER_HEADLESS = True
GERAPY_PYPPETEER_DUMPIO = False
GERAPY_PYPPETEER_DEVTOOLS = False
GERAPY_PYPPETEER_EXECUTABLE_PATH = None
GERAPY_PYPPETEER_DISABLE_EXTENSIONS = True
GERAPY_PYPPETEER_HIDE_SCROLLBARS = True
GERAPY_PYPPETEER_MUTE_AUDIO = True
GERAPY_PYPPETEER_NO_SANDBOX = True
GERAPY_PYPPETEER_DISABLE_SETUID_SANDBOX = True
GERAPY_PYPPETEER_DISABLE_GPU = True
Example
For more detail, please see example.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
gerapy-pyppeteer-0.0.3.tar.gz
(5.1 kB
view hashes)
Built Distribution
Close
Hashes for gerapy_pyppeteer-0.0.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0000b55a6923eeda319e1a15e31b032fe7fe6568d8da6f2dd85e35c602d1e13 |
|
MD5 | 4861b0b89e30b283e20cbff0f7d5aa9f |
|
BLAKE2b-256 | 51bba3384b6b9cd35b74f98afa9dcfce862e722b979a6be4ec55398f37397c06 |