async-site-crawler

A high-performance, concurrent web crawler built in Python that recursively crawls websites and exports structured data to CSV reports.

Features

• Async/Concurrent Crawling: Uses asyncio and aiohttp for fast, non-blocking HTTP requests
• Configurable Limits: Control max concurrent requests and total pages to crawl
• Smart URL Handling: Normalizes URLs to avoid duplicate crawls
• Same-Domain Filtering: Stays within the target website domain
• HTML Parsing: Extracts h1 tags, paragraphs, links, and images using BeautifulSoup
• CSV Export: Generates structured reports for easy analysis
• Graceful Stopping: Cancels in-flight tasks when limits are reached
• Error Handling: Handles timeouts, non-HTML content, and network failures

Installation

pip install async-site-crawler

Or with uv:

uv add async-site-crawler

Usage

Command Line

async-site-crawler https://example.com

With custom settings:

async-site-crawler https://example.com 10 50

As a Python Library

import asyncio
from async_site_crawler import crawl_site_async, write_csv_report

async def main():
    page_data = await crawl_site_async(
        base_url="https://example.com",
        max_concurrency=5,
        max_pages=20
    )
    write_csv_report(page_data, filename="report.csv")

asyncio.run(main())

Output

The crawler generates a report.csv file with the following columns:

• page_url: The URL of the crawled page
• h1: The main heading (h1 tag) content
• first_paragraph: Text from the first paragraph (prioritizes <main> tag)
• outgoing_link_urls: All links found on the page (semicolon-separated)
• image_urls: All image URLs found on the page (semicolon-separated)

License

This project is open source and available under the MIT License.

Author

Utkarsh736