Skip to main content

A Scrapy Middleware to bypass the CloudFlare's anti-bot protection

Project description

[![PyPI](https://img.shields.io/pypi/v/scrapy_cloudflare_middleware.svg)](https://pypi.python.org/pypi/scrapy_cloudflare_middleware)

## Scrapy "CloudFlare" middleware

A Scrapy Middleware to bypass the CloudFlare's anti-bot protection, based on [cloudflare-scrape](https://github.com/Anorov/cloudflare-scrape).

### Installation
```
pip install scrapy_cloudflare_middleware
```

### Usage

Add the middleware to your `DOWNLOADER_MIDDLEWARES` settings

```python
DOWNLOADER_MIDDLEWARES = {
'scraping_hub.middlewares.CloudFlareMiddleware': 560
}
```
The priority (560) is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.

Done. Happy scraping !








Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapy-cloudflare-middleware, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size scrapy_cloudflare_middleware-0.0.1-py3-none-any.whl (3.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size scrapy_cloudflare_middleware-0.0.1.tar.gz (2.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page