Skip to main content

Pyppeteer Components for Scrapy & Gerapy

Project description

Gerapy Pyppeteer

This is a package for supporting pyppeteer in Scrapy, also this package is a module in Gerapy.


pip3 install gerapy-pyppeteer


You can use PyppeteerRequest to specify a request which uses pyppeteer to render.

For example:

yield PyppeteerRequest(detail_url, callback=self.parse_detail)

And you also need to enable PyppeteerMiddleware in DOWNLOADER_MIDDLEWARES:

    'gerapy_pyppeteer.downloadermiddlewares.PyppeteerMiddleware': 543,

Congratulate, you've finished the all of the required configuration.

If you run the Spider again, Pyppeteer will be started to render every web page which you configured the request as PyppeteerRequest.


GerapyPyppeteer provides some optional settings.


You can directly use Scrapy's setting to set Concurrency of Pyppeteer, for example:


Pretend as Real Browser

Some website will detect WebDriver or Headless, GerapyPyppeteer can pretend Chromium by inject scripts. This is enabled by default.

You can close it if website does not detect WebDriver to speed up:


Also you can use pretend attribute in PyppeteerRequest to overwrite this configuration.

Logging Level

By default, Pyppeteer will log all the debug messages, so GerapyPyppeteer configured the logging level of Pyppeteer to WARNING.

If you want to see more logs from Pyppeteer, you can change the this setting:

import logging

Download Timeout

Pyppeteer may take some time to render the required web page, you can also change this setting, default is 30s:

# pyppeteer timeout


By default, Pyppeteer is running in Headless mode, you can also change it to False as you need, default is True:


Window Size

You can also set the width and height of Pyppeteer window:


Default is 1400, 700.

Pyppeteer Args

You can also change the args of Pyppeteer, such as dumpio, devtools, etc.

Optional settings and their default values:


Disable loading of specific resource type

You can disable the loading of specific resource type to decrease the loading time of web page. You can configure the disabled resource types using GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES:


For example, if you want to disable the loading of css and javascript, you can set as below:


All of the optional resource type list:

  • document: the Original HTML document
  • stylesheet: CSS files
  • script: JavaScript files
  • image: Images
  • media: Media files such as audios or videos
  • font: Fonts files
  • texttrack: Text Track files
  • xhr: Ajax Requests
  • fetch: Fetch Requests
  • eventsource: Event Source
  • websocket: Websocket
  • manifest: Manifest files
  • other: Other files


You can get screenshot of loaded page, you can pass screenshot args to PyppeteerRequest as dict:

  • type (str): Specify screenshot type, can be either jpeg or png. Defaults to png.
  • quality (int): The quality of the image, between 0-100. Not applicable to png image.
  • fullPage (bool): When true, take a screenshot of the full scrollable page. Defaults to False.
  • clip (dict): An object which specifies clipping region of the page. This option should have the following fields:
    • x (int): x-coordinate of top-left corner of clip area.
    • y (int): y-coordinate of top-left corner of clip area.
    • width (int): width of clipping area.
    • height (int): height of clipping area.
  • omitBackground (bool): Hide default white background and allow capturing screenshot with transparency.
  • encoding (str): The encoding of the image, can be either base64 or binary. Defaults to binary. If binary it will return BytesIO object.

For example:

yield PyppeteerRequest(start_url, callback=self.parse_index, wait_for='.item .name', screenshot={
            'type': 'png',
            'fullPage': True

then you can get screenshot result in response.meta['screenshot']:

Simplest save it to file:

def parse_index(self, response):
    with open('screenshot.png', 'wb') as f:

If you want to enable screenshot for all requests, you can configure it by GERAPY_PYPPETEER_SCREENSHOT.

For example:

    'type': 'png',
    'fullPage': True


PyppeteerRequest provide args which can override global settings above.

  • url: request url
  • callback: callback
  • one of "load", "domcontentloaded", "networkidle0", "networkidle2". see, default is domcontentloaded
  • wait_for: wait for some element to load, also supports dict
  • script: script to execute
  • proxy: use proxy for this time, like http://x.x.x.x:x
  • sleep: time to sleep after loaded, override GERAPY_PYPPETEER_SLEEP
  • timeout: load timeout, override GERAPY_PYPPETEER_DOWNLOAD_TIMEOUT
  • ignore_resource_types: ignored resource types, override GERAPY_PYPPETEER_IGNORE_RESOURCE_TYPES
  • pretend: pretend as normal browser, override GERAPY_PYPPETEER_PRETEND
  • screenshot: ignored resource types, see, override GERAPY_PYPPETEER_SCREENSHOT

For example, you can configure PyppeteerRequest as:

from gerapy_pyppeteer import PyppeteerRequest

def parse(self, response):
    yield PyppeteerRequest(url, 
        script='() => { console.log(document) }',

Then Pyppeteer will:

  • wait for document to load
  • wait for title to load
  • execute console.log(document) script
  • sleep for 2s
  • return the rendered web page content


For more detail, please see example.

Also you can directly run with Docker:

docker run germey/gerapy-pyppeteer-example


2020-07-13 01:49:13 [scrapy.utils.log] INFO: Scrapy 2.2.0 started (bot: example)
2020-07-13 01:49:13 [scrapy.utils.log] INFO: Versions: lxml, libxml2 2.9.9, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.7 (default, May  6 2020, 04:59:01) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Darwin-19.4.0-x86_64-i386-64bit
2020-07-13 01:49:13 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2020-07-13 01:49:13 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'example',
 'NEWSPIDER_MODULE': 'example.spiders',
 'RETRY_HTTP_CODES': [403, 500, 502, 503, 504],
 'SPIDER_MODULES': ['example.spiders']}
2020-07-13 01:49:13 [scrapy.extensions.telnet] INFO: Telnet Password: 83c276fb41754bd0
2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled extensions:
2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled spider middlewares:
2020-07-13 01:49:13 [scrapy.middleware] INFO: Enabled item pipelines:
2020-07-13 01:49:13 [scrapy.core.engine] INFO: Spider opened
2020-07-13 01:49:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-07-13 01:49:13 [scrapy.extensions.telnet] INFO: Telnet console listening on
2020-07-13 01:49:13 [] INFO: crawling
2020-07-13 01:49:13 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:13 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:14 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:19 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: close pyppeteer
2020-07-13 01:49:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET> (referer: None)
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:20 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:21 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: close pyppeteer
2020-07-13 01:49:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET> (referer:
2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:24 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:25 [scrapy.core.scraper] DEBUG: Scraped from <200>
{'name': '壁穴ヘブンホール',
 'score': '5.6',
 'tags': ['BL漫画', '小基漫', 'BL', '『又腐又基』', 'BLコミック']}
2020-07-13 01:49:25 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
2020-07-13 01:49:25 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: close pyppeteer
2020-07-13 01:49:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET> (referer:
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: processing request <GET>
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: set options {'headless': True, 'dumpio': False, 'devtools': False, 'args': ['--window-size=1400,700', '--disable-extensions', '--hide-scrollbars', '--mute-audio', '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu']}
2020-07-13 01:49:26 [scrapy.core.scraper] DEBUG: Scraped from <200>
{'name': '冒险小虎队', 'score': '9.4', 'tags': ['冒险小虎队', '童年', '冒险', '推理', '小时候读的']}
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: waiting for .item .name finished
2020-07-13 01:49:26 [gerapy.pyppeteer] DEBUG: crawling
2020-07-13 01:49:27 [gerapy.pyppeteer] DEBUG: wait for .item .name finished
2020-07-13 01:49:27 [gerapy.pyppeteer] DEBUG: close pyppeteer

Trouble Shooting

Pyppeteer does not start properly

Chromium download speed is too slow, it can not be used normally.

Here are two solutions:

Solution 1 (Recommended)

Modify drive download source at pyppeteer/ line 22:

# Default:
# modify

Solution 2

Modify drive execution path at pyppeteer/ line 45:

# Default:
chromiumExecutable = {
    'linux': DOWNLOADS_FOLDER / REVISION / 'chrome-linux' / 'chrome',
    'mac': (DOWNLOADS_FOLDER / REVISION / 'chrome-mac' / '' /
            'Contents' / 'MacOS' / 'Chromium'),
    'win32': DOWNLOADS_FOLDER / REVISION / windowsArchive / 'chrome.exe',
    'win64': DOWNLOADS_FOLDER / REVISION / windowsArchive / 'chrome.exe',

You can find your own operating system, modify your chrome or chrome executable path.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gerapy-pyppeteer-0.0.13.tar.gz (18.0 kB view hashes)

Uploaded source

Built Distribution

gerapy_pyppeteer-0.0.13-py2.py3-none-any.whl (13.4 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page