Python asynchronous library for web scraping
Project description
Python async library for web scrapping
Installing
pip install aioscrapy
Usage
from aioscrapy import Client, WebClient, SingleSessionPool, Dispatcher, SimpleWorker
class CustomClient(Client[str, dict]):
def __init__(self, client: WebClient):
self._client = client
async def fetch(self, key: str) -> dict:
data = self._client.fetch(key)
# Processing data, getting result
return {"result": "some_result"}
async def main():
pool = SingleSessionPool()
dispatcher = Dispatcher(['https://start-url.example.com'])
client = CustomClient(WebClient(pool))
worker = SimpleWorker(dispatcher, client)
result = await worker.run()
return result
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aioscrapy-0.1.3.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for aioscrapy-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a814b94ce976f7bc8337555f437760f89ff60d867e515fe792a715eba9fcdc0 |
|
MD5 | 06f83604ca425d798520eccad6955448 |
|
BLAKE2b-256 | 8c2ed8afacabfd7c762c39387a177c2e288f2321946b318f859539f5462d0873 |