A package designed to scrape webpages using aiohttp and asyncio. Has some error handling to overcome common issues such as sites blocking you after n requests over a short period.
Project description
Async-scrape
Perform webscrape asyncronously
Async-scrape is a package which uses asyncio and aiohttp to scrape websites and has useful features built in.
Features
- Breaks - pause scraping when a website blocks your requests consistently
- Rate limit - slow down scraping to prevent being blocked
Installation
Async-scrape requires C++ Build tools v15+ to run.
pip install async-scrape
How to use it
#Create an instance
from async_scrape import AsyncScrape
def post_process(resp):
"""Function to process the gathered response from the request"""
if resp.status == 200:
return "Request worked"
else:
return "Request failed"
async_Scrape = AsyncScrape(
post_process_func=post_process,
post_process_kwargs={},
fetch_error_handler=None,
use_proxy=False,
proxy=None,
pac_url=None,
acceptable_error_limit=100,
attempt_limit=5,
rest_between_attempts=True,
rest_wait=60
)
urls = [
"https://www.google.com",
"https://www.bing.com",
]
resps = async_Scrape.scrape_all(urls)
Response object is a list of dicts in the format:
{
"url":url, #url of request
"func_resp":func_resp, #response from post processing function
"status":resp.status, #http status
"error":None #any error encountered
}
License
MIT
Free Software, Hell Yeah!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
async-scrape-0.1.6.tar.gz
(7.7 kB
view hashes)
Built Distribution
Close
Hashes for async_scrape-0.1.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 396c87c8dedd0753e5b801f6c2b5d5b207075b513955c83e83c10fe760a4556d |
|
MD5 | 87fafa6a2cd22bbe205c0c3b4c711774 |
|
BLAKE2b-256 | 48dc3f3b0912f011610293c1ea737205be65962162e6e50481f815a9c7c54b9d |