Skip to main content

JavaScript support and proxy rotation for Scrapy with ScrapingBee

Project description

Scrapy ScrapingBee Middleware

build version python

Integrate Scrapy with ScrapingBee API to use headless browsers for JavaScript and proxy rotation. Requires to create an account on to get an API key.


pip install scrapy-scrapingbee


Add your SCRAPINGBEE_API_KEY and the ScrapingBeeMiddleware to your project Don't forget to set CONCURRENT_REQUESTS according to your ScrapingBee plan.


    'scrapy_scrapingbee.ScrapingBeeMiddleware': 725,



Inherit your spiders from ScrapingBeeSpider and yield a ScrapingBeeRequest.

ScrapingBeeSpider overrides the default logger to hide your API key in the Scrapy logs.

Below you can see an example from the spider in

from scrapy_scrapingbee import ScrapingBeeSpider, ScrapingBeeRequest

JS_SNIPPET = 'window.scrollTo(0, document.body.scrollHeight);'

class HttpbinSpider(ScrapingBeeSpider):
    name = 'httpbin'
    start_urls = [

    def start_requests(self):
        for url in self.start_urls:
            yield ScrapingBeeRequest(url, params={
                # 'render_js': False,
                # 'block_ads': True,
                # 'block_resources': False,
                # 'js_snippet': JS_SNIPPET,
                # 'premium_proxy': True,
                # 'country_code': 'fr',
                # 'return_page_source': True,
                # 'wait': 3000,
                # 'wait_for': '#swagger-ui',
                # 'Accept-Language': 'En-US',
                # 'name_1': 'value_1',

    def parse(self, response):

You can pass ScrapingBee parameters in the params argument of a ScrapingBeeRequest. Headers and cookies are passed like a normal Scrapy Request. ScrapingBeeRequest formats all parameters, headers and cookies to the format expected by the ScrapingBee API.


Add your API key to

To run the examples you need to clone this repository. In your terminal, go to examples/httpbin/httpbin and run the example spider with:

scrapy crawl httpbin

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-scrapingbee-0.0.5.tar.gz (4.2 kB view hashes)

Uploaded Source

Built Distribution

scrapy_scrapingbee-0.0.5-py3-none-any.whl (5.1 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page