A scalable headless data fetching library written with python and message queue service to enable quickly and easily prasing web data in a distributive way.
# pifetcher A scalable headless data fetching library written with python and message queue service to enable quickly and easily parsing web data in a distributive way.
## To install `bash pip install pifetcher `
## PYPI Link [https://pypi.org/project/pifetcher/]
## dependencies: - pyppeteer - BeautifulSoup4 - boto3
event-callback-based interaction between user defined logic and the pre-disigned fetch worker
process works in batches, library user will be able to capture the event of a batch of works have been finished
easy to use, only needs to inherit the FetchWorker class and implement the basic call back functions
it’s design to use message queue, enbles more than just one worker to perform data fetching in order to scale the application
## how to use:
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.