Efficient asyncio based web crawler
Project description
Features
- Asynchronous downloading using aiohttp
- All downloads cached locally in sqlite
- Continue an interrupted crawl
Example
>>> import ws ...
Install
Install from pypi:
pip install ws
Or checkout latest version from repository:
hg clone https://bitbucket.org/richardpenman/ws