Asynchronous web crawler built on asyncio
Project description
AIOCrawler
Asynchronous web crawler built on asyncio
Installation
pip install pyaiocrawler
Usage
Generating sitemap
from aiocrawler import AIOCrawler
crawler = AIOCrawler('https://www.google.com', depth=3)
sitemap = await crawler.generate_sitemap()
Configuring the crawler
from aiocrawler import AIOCrawler
crawler = AIOCrawler(
init_url='https://www.google.com',
depth=3,
concurrency=300,
user_agent='My Amazing Crawler'
)
Extending the crawler
WIP
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyaiocrawler-0.1.2.tar.gz
(3.2 kB
view hashes)
Built Distribution
Close
Hashes for pyaiocrawler-0.1.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4f69c992226aaef0b19e6c2432d7e9a5d0a452cf050a8d8afb78371b3e47436f |
|
| MD5 | 11ac0f15dd16ec6124eae2b1d118004a |
|
| BLAKE2b-256 | 73bc9d688727de942cf7201a7c561cd5c08fca56866ab714b43d26bd740d4e1a |
