Cloudflare scraper and cralwer written in Async
Project description
cfcrawler
Cloudflare scraper and cralwer written in Async, In-place library for HTTPX. Crawl website that has cloudflare enabled, easier than ever!
- Github repository: https://github.com/ManiMozaffar/cfcrawler/
Getting started
To use library, simply replace your aiohttp client with ours!
from cfcrawler import AsyncClient
async def get(url):
client = AsyncClient()
await client.get(url)
You can also rotate user agents
from cfcrawler import AsyncClient
client = AsyncClient()
client.rotate_useragent()
You can also specify which browser you want to use
from cfcrawler.types import Browser
from cfcrawler import AsyncClient
AsyncClient(browser=Browser.CHROME)
You can also use asyncer to syncify the implementation
from cfcrawler import AsyncClient
from asyncer import syncify
def get(url):
client = AsyncClient()
syncify(client.get)(url)
Coming Next
- CF JS Challenge solver
- Captcha solver integration (2Captcha and etc)
Contribution
I'll work on this library in few months, I don't have free time right now, but feel free to contribute. I'll check and test the PRs myself!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cfcrawler-0.0.2.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for cfcrawler-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1baa0dec19a43a4588e67e424c51bb9052a66dfb3f6e43d698420a21d2780eae |
|
MD5 | 1167b730d856955259fa772465a4023d |
|
BLAKE2b-256 | a50f57b9767838367a7c6a542a3b4c4078ccf2d9f59a7aad8f3227a0c6fe62ed |