py-websites-scraper 0.1.0

Mutliple we scraper using aiohttp to make it easy to scrape multiple URLs

About

Scrape main content on multiple websites using Python in parallel.

Dependency

• AsyncIO
• aiohttp
• Readability-lxml

How to use

pip install py-web-scraper

Quick usage:

import asyncio
from py_websites_scraper import scrape_urls

urls = ["https://news.ycombinator.com", "https://example.com"]
data = asyncio.run(scrape_urls(urls, max_concurrency=5))
for item in data:
    print(item["url"], item.get("title"))

You can add any parameters for aiohttp to perform the request like headers, proxy, and more.

Example:

urls = []
 results = await scrape_urls(
        urls,
        proxy="YOUR_PROXY_INFO",
        headers={"User-Agent": "USER_AGENT_INFO"},
    )

Limitation

• Gated content
• Dynamic generated content

How the test the package locally for Dev

Install in editable mode:

pip install -e .

Run any file that importing this package

python test_local.py