Asyncio based web crawling framework
Project description
Features
- Asynchronous downloading using aiohttp
- Downloads cached locally in sqlite
- Continue an interrupted crawl
- Proxies
- Cookies
- Handle redirects
- Retry 5XX errors
Example
>>> import asyncrawler ...
Install
Install from pypi:
pip install asyncrawler
Or checkout latest version from repository:
git clone https://github.com/richardpenman/asyncrawler
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size | File type | Python version | Upload date | Hashes |
|---|---|---|---|---|
| Filename, size asyncrawler-0.2.1.tar.gz (9.6 kB) | File type Source | Python version None | Upload date | Hashes View |
