Asyncio based web crawling framework
Project description
Features
Asynchronous downloading using aiohttp
Downloads cached locally in sqlite
Continue an interrupted crawl
Proxies
Cookies
Handle redirects
Retry 5XX errors
Example
>>> import asyncrawler
...
Install
Install from pypi:
pip install asyncrawler
Or checkout latest version from repository:
git clone https://github.com/richardpenman/asyncrawler
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
asyncrawler-0.2.1.tar.gz
(9.6 kB
view hashes)