Skip to main content

scrapy的一个下载中间件,绕过cloudflare检测

Project description

AroayCloudScraper

scrapy一个插件,绕过cloudflare检测,主要是封装 cloudscraper模块,并且将cloudscraper模块在scrapy中异步执行

使用,在setting添加

DOWNLOADER_MIDDLEWARES = {
    'aroay_cloudscraper.downloadermiddlewares.CloudScraperMiddleware': 543,
}

需setting设置

# 默认日志级别
AROAY_CLOUDSCRAPER_LOGGING_LEVEL = logging.DEBUG

默认超时
AROAY_CLOUDSCRAPER_DOWNLOAD_TIMEOUT = 30

# 默认延迟
AROAY_CLOUDSCRAPER_DELAY = 1

#必须设置,否则报错
COMPRESSION_ENABLED = False

RETRY_ENABLED: True
RETRY_TIMES: 3

代理使用

    def start_requests(self):
        for page in range(1, 2):
            yield CloudScraperRequest(self.base_url, callback=self.parse_index, dont_filter=True, proxy={
                "http": "http://username:password@ip:port",
                "https": "http://username:password@ip:port",
            },cookies={"over18":"1"},timeout=5)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aroay_cloudscraper-1.5.tar.gz (4.1 kB view details)

Uploaded Source

Built Distribution

aroay_cloudscraper-1.5-py3-none-any.whl (5.9 kB view details)

Uploaded Python 3

File details

Details for the file aroay_cloudscraper-1.5.tar.gz.

File metadata

  • Download URL: aroay_cloudscraper-1.5.tar.gz
  • Upload date:
  • Size: 4.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.4

File hashes

Hashes for aroay_cloudscraper-1.5.tar.gz
Algorithm Hash digest
SHA256 7e5a42f01757d1373dd484cfa3b23a7b4510252c43706ad40134622eedda380e
MD5 8609637a236305c73e028c2bddf5ca97
BLAKE2b-256 0bb14fefe3bd12c96f351f8e4f8bceca8b83d7e618a6a747dfed44542d3ef88a

See more details on using hashes here.

File details

Details for the file aroay_cloudscraper-1.5-py3-none-any.whl.

File metadata

  • Download URL: aroay_cloudscraper-1.5-py3-none-any.whl
  • Upload date:
  • Size: 5.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.4

File hashes

Hashes for aroay_cloudscraper-1.5-py3-none-any.whl
Algorithm Hash digest
SHA256 3fece153fde04b3f90591bfd827b644659c97c5633b1ce73307b6fa622666960
MD5 1050a82dbeebe85738f4330f3f194386
BLAKE2b-256 eb7f90b74713523e6feb38ffb1db47130832a593f31f85c05de233de3f41c9a5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page