Asyncio based web crawling framework
- Asynchronous downloading using aiohttp
- Downloads cached locally in sqlite
- Continue an interrupted crawl
- Handle redirects
- Retry 5XX errors
>>> import asyncrawler ...
Install from pypi:
pip install asyncrawler
Or checkout latest version from repository:
git clone https://github.com/richardpenman/asyncrawler
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.