scrapy的一个下载中间件,绕过cloudflare检测
Project description
AroayCloudScraper
scrapy一个插件,绕过cloudflare检测,主要是封装 cloudscraper模块
需setting设置
# 默认日志级别
AROAY_CLOUDSCRAPER_LOGGING_LEVEL = logging.DEBUG
默认超时
AROAY_CLOUDSCRAPER_DOWNLOAD_TIMEOUT = 30
# 默认延迟
AROAY_CLOUDSCRAPER_DELAY = 1
#必须设置,否则报错
COMPRESSION_ENABLED = False
代理使用
def start_requests(self):
for page in range(1, 2):
yield CloudScraperRequest(self.base_url, callback=self.parse_index, dont_filter=True, proxy={
"http": "http://username:password@ip:port",
"https": "http://username:password@ip:port",
},cookies={"over18":"1"},timeout=5)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aroay_cloudscraper-1.2.tar.gz
(4.9 kB
view hashes)
Built Distribution
Close
Hashes for aroay_cloudscraper-1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d328e97a1e870929ecc58c880146ea49f738fd63e3b802fbe982272e19baeb65 |
|
MD5 | 4c92215a4c3787ab8f2e7310e465e472 |
|
BLAKE2b-256 | 875579b6d223c54d118449f6506a2071bfd8cbcab60c274b432e06018db18c27 |