Asynchronous web crawler built on asyncio
Project description
AIOCrawler
Asynchronous web crawler built on asyncio
Installation
pip install pyaiocrawler
Usage
Generating sitemap
from aiocrawler import AIOCrawler
crawler = AIOCrawler('https://www.google.com', depth=3)
sitemap = await crawler.generate_sitemap()
Configuring the crawler
from aiocrawler import AIOCrawler
crawler = AIOCrawler(
init_url='https://www.google.com',
depth=3,
concurrency=300,
user_agent='My Amazing Crawler'
)
Extending the crawler
WIP
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyaiocrawler-0.1.1.tar.gz
(3.2 kB
view hashes)
Built Distribution
Close
Hashes for pyaiocrawler-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa8c555cd308ff5abe5febc76782258d9c5e69c3f17b3651f1f8e86666849d6f |
|
MD5 | 8ae12f9d8326d18e1d4e685b775ed924 |
|
BLAKE2b-256 | ad04b182f0f8b250f38a650f12b939d57a1f2a5d0d84c8847eb0fb7352d41ca3 |