A simple Python module to bypass Cloudflare"s anti-bot page. See https://github.com/Anorov/cloudflare-scrape for more information.

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for Anorov from gravatar.com Anorov

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • Author: Anorov
  • Tags cloudflare, scraping
Report project as malware

Project description

cloudflare-scrape

A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with Requests. Cloudflare changes their techniques periodically, so I will update this repo frequently.

This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports Javascript, though they may add additional techniques in the future.

Due to Cloudflare continually changing and hardening their protection page, cloudflare-scrape requires Node.js to solve Javascript challenges. This allows the script to easily impersonate a regular web browser without explicitly deobfuscating and parsing Cloudflare's Javascript.

Note: This only works when regular Cloudflare anti-bots is enabled (the "Checking your browser before accessing..." loading page). If there is a reCAPTCHA challenge, you're out of luck. Thankfully, the Javascript check page is much more common.

For reference, this is the default message Cloudflare uses for these sorts of pages:

Checking your browser before accessing website.com.

This process is automatic. Your browser will redirect to your requested content shortly.

Please allow up to 5 seconds...

Any script using cloudflare-scrape will sleep for 5 seconds for the first visit to any site with Cloudflare anti-bots enabled, though no delay will occur after the first request.

Installation

Simply run pip install cfscrape. The PyPI package is at https://pypi.python.org/pypi/cfscrape/

Alternatively, clone this repository and run python setup.py install.

Dependencies

  • Python 2.6 - 3.x
  • Requests >= 2.0
  • Node.js
    • Your computer or server may already have it (check with node -v). If not, you can install it with apt-get install nodejs on Ubuntu. Debian requires nodejs-legacy. Otherwise, please read Node's installation instructions.

python setup.py install will install the Python dependencies automatically. Node is the only application you need to install yourself.

Updates

Cloudflare modifies their anti-bot protection page occasionally. So far it has changed maybe once per year on average.

If you notice that the anti-bot page has changed, or if this module suddenly stops working, please create a GitHub issue so that I can update the code accordingly.

  • Many issues are a result of users not updating to the latest release of this project. Before filing an issue, please run the following command:
pip show cfscrape

If the value of the version field is not the latest release, please run the following to update your package:

pip install cfscrape -U

If you are still encountering a problem, open an issue and please include:

  • The full exception and stack trace.
  • The URL of the Cloudflare-protected page which the script does not work on.
  • A Pastebin or Gist containing the HTML source of the protected page.
  • The version number from pip show cfscrape.

This issue comment is a good example.

Usage

The simplest way to use cloudflare-scrape is by calling create_scraper().

import cfscrape

scraper = cfscrape.create_scraper()  # returns a CloudflareScraper instance
# Or: scraper = cfscrape.CloudflareScraper()  # CloudflareScraper inherits from requests.Session
print scraper.get("http://somesite.com").content  # => "<!DOCTYPE html><html><head>..."

That's it. Any requests made from this session object to websites protected by Cloudflare anti-bot will be handled automatically. Websites not using Cloudflare will be treated normally. You don't need to configure or call anything further, and you can effectively treat all websites as if they're not protected with anything.

You use cloudflare-scrape exactly the same way you use Requests. (CloudflareScraper works identically to a Requests Session object.) Just instead of calling requests.get() or requests.post(), you call scraper.get() or scraper.post(). Consult Requests' documentation for more information.

Options

Existing session

If you already have an existing Requests session, you can pass it to create_scraper() to continue using that session.

session = requests.session()
session.headers = ...
scraper = cfscrape.create_scraper(sess=session)

Unfortunately, not all of Requests' session attributes are easily transferable, so if you run into problems with this, you should replace your initial sess = requests.session() call with sess = cfscrape.create_scraper().

Delays

Normally, when a browser is faced with a Cloudflare IUAM challenge page, Cloudflare requires the browser to wait 5 seconds before submitting the challenge answer. If a website is under heavy load, sometimes this may fail. One solution is to increase the delay (perhaps to 10 or 15 seconds, depending on the website). If you would like to override this delay, pass the delay keyword argument to create_scraper() or CloudflareScraper().

There is no need to override this delay unless cloudflare-scrape generates an error recommending you increase the delay.

scraper = cfscrape.create_scraper(delay=10)

Integration

It's easy to integrate cloudflare-scrape with other applications and tools. Cloudflare uses two cookies as tokens: one to verify you made it past their challenge page and one to track your session. To bypass the challenge page, simply include both of these cookies (with the appropriate user-agent) in all HTTP requests you make.

To retrieve just the cookies (as a dictionary), use cfscrape.get_tokens(). To retrieve them as a full Cookie HTTP header, use cfscrape.get_cookie_string().

get_tokens and get_cookie_string both accept Requests' usual keyword arguments (like get_tokens(url, proxies={"http": "socks5://localhost:9050"})). Please read Requests' documentation on request arguments for more information.

User-Agent Handling

The two integration functions return a tuple of (cookie, user_agent_string). You must use the same user-agent string for obtaining tokens and for making requests with those tokens, otherwise Cloudflare will flag you as a bot. That means you have to pass the returned user_agent_string to whatever script, tool, or service you are passing the tokens to (e.g. curl, or a specialized scraping tool), and it must use that passed user-agent when it makes HTTP requests.

If your tool already has a particular user-agent configured, you can make cloudflare-scrape use it with cfscrape.get_tokens("http://somesite.com/", user_agent="User-Agent Here") (also works for get_cookie_string). Otherwise, a randomly selected user-agent will be used.

Integration examples

Remember, you must always use the same user-agent when retrieving or using these cookies. These functions all return a tuple of (cookie_dict, user_agent_string).

Retrieving a cookie dict through a proxy

get_tokens is a convenience function for returning a Python dict containing Cloudflare's session cookies. For demonstration, we will configure this request to use a proxy. (Please note that if you request Cloudflare clearance tokens through a proxy, you must always use the same proxy when those tokens are passed to the server. Cloudflare requires that the challenge-solving IP and the visitor IP stay the same.)

If you do not wish to use a proxy, just don't pass the proxies keyword argument. These convenience functions support all of Requests' normal keyword arguments, like params, data, and headers.

import cfscrape

proxies = {"http": "http://localhost:8080", "https": "http://localhost:8080"}
tokens, user_agent = cfscrape.get_tokens("http://somesite.com", proxies=proxies)
print tokens
# => {'cf_clearance': 'c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600', '__cfduid': 'dd8ec03dfdbcb8c2ea63e920f1335c1001426733158'}

Retrieving a cookie string

get_cookie_string is a convenience function for returning the tokens as a string for use as a Cookie HTTP header value.

This is useful when crafting an HTTP request manually, or working with an external application or library that passes on raw cookie headers.

import cfscrape
request = "GET / HTTP/1.1\r\n"

cookie_value, user_agent = cfscrape.get_cookie_string("http://somesite.com")
request += "Cookie: %s\r\nUser-Agent: %s\r\n" % (cookie_value, user_agent)

print request

# GET / HTTP/1.1\r\n
# Cookie: cf_clearance=c8f913c707b818b47aa328d81cab57c349b1eee5-1426733163-3600; __cfduid=dd8ec03dfdbcb8c2ea63e920f1335c1001426733158
# User-Agent: Some/User-Agent String

curl example

Here is an example of integrating cloudflare-scrape with curl. As you can see, all you have to do is pass the cookies and user-agent to curl.

import subprocess
import cfscrape

# With get_tokens() cookie dict:

# tokens, user_agent = cfscrape.get_tokens("http://somesite.com")
# cookie_arg = "cf_clearance=%s; __cfduid=%s" % (tokens["cf_clearance"], tokens["__cfduid"])

# With get_cookie_string() cookie header; recommended for curl and similar external applications:

cookie_arg, user_agent = cfscrape.get_cookie_string("http://somesite.com")

# With a custom user-agent string you can optionally provide:

# ua = "Scraping Bot"
# cookie_arg, user_agent = cfscrape.get_cookie_string("http://somesite.com", user_agent=ua)

result = subprocess.check_output(["curl", "--cookie", cookie_arg, "-A", user_agent, "http://somesite.com"])

Trimmed down version. Prints page contents of any site protected with Cloudflare, via curl. (Warning: shell=True can be dangerous to use with subprocess in real code.)

url = "http://somesite.com"
cookie_arg, user_agent = cfscrape.get_cookie_string(url)
cmd = "curl --cookie {cookie_arg} -A {user_agent} {url}"
print(subprocess.check_output(cmd.format(cookie_arg=cookie_arg, user_agent=user_agent, url=url), shell=True))

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for Anorov from gravatar.com Anorov

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • Author: Anorov
  • Tags cloudflare, scraping

Release history Release notifications | RSS feed

2.1.1

2.1.0

2.0.8

2.0.7

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

This version

2.0.0

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.1

1.8.0

1.7.1

1.7.0

1.6.8

1.6.7

1.6.6

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.4.3

1.4.2

1.4.1

1.4

1.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cfscrape-2.0.0.tar.gz (10.9 kB view details)

Uploaded Source

Built Distributions

cfscrape-2.0.0-py27-none-any.whl (10.4 kB view details)

Uploaded Python 2.7

cfscrape-2.0.0-py3.7.egg (14.3 kB view details)

Uploaded Egg

cfscrape-2.0.0-py3-none-any.whl (10.4 kB view details)

Uploaded Python 3

File details

Details for the file cfscrape-2.0.0.tar.gz.

File metadata

  • Download URL: cfscrape-2.0.0.tar.gz
  • Upload date:
  • Size: 10.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for cfscrape-2.0.0.tar.gz
Algorithm Hash digest
SHA256 a65608d830682b37e6725853a869e5c94f1cd41b8c4b9dec6ebc8cee1563a0c6
MD5 8bd1f57a175168b4d0bbe82cfd053f08
BLAKE2b-256 5e3b57458e3c9dc53e6c8006600a22d1f3e4917b3dfe952c2333fd0165426208

See more details on using hashes here.

File details

Details for the file cfscrape-2.0.0-py27-none-any.whl.

File metadata

  • Download URL: cfscrape-2.0.0-py27-none-any.whl
  • Upload date:
  • Size: 10.4 kB
  • Tags: Python 2.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for cfscrape-2.0.0-py27-none-any.whl
Algorithm Hash digest
SHA256 7f02cd53beda4b426810ccfa74f2198e20500591964020ee9f48367e43cd77c4
MD5 fb3308e4d7c29375d5626199b3ec05a2
BLAKE2b-256 9d713fb05468e3be198d4bc4fc9aa752416ffb574faa5de578a11850b96c59fd

See more details on using hashes here.

File details

Details for the file cfscrape-2.0.0-py3.7.egg.

File metadata

  • Download URL: cfscrape-2.0.0-py3.7.egg
  • Upload date:
  • Size: 14.3 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for cfscrape-2.0.0-py3.7.egg
Algorithm Hash digest
SHA256 07a70f3acdc2ad681e2050581004c108b9e3a78ef7a18a2c18f299d429b554be
MD5 726780b22f9138640e434ed16d2c5f4e
BLAKE2b-256 62f412473eb864a441e314e7ee888f2218cbf4be3b809036ba315da1c93923ca

See more details on using hashes here.

File details

Details for the file cfscrape-2.0.0-py3-none-any.whl.

File metadata

  • Download URL: cfscrape-2.0.0-py3-none-any.whl
  • Upload date:
  • Size: 10.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for cfscrape-2.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 08da77558391d77578bcf851006fb1412c2398c3e886b7bfd8b19ba04a120e7e
MD5 cc7905b0f21499edf42c3938b464e899
BLAKE2b-256 9c39cf549e87333b6c2d784264ddb0e15db49e764ad9f9661b94c419dfbb422d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page