A scalable headless data fetching library written with python and message queue service to enable quickly and easily prasing web data in a distributive way.
Project description
# pifetcher A scalable headless data fetching library written with python and message queue service to enable quickly and easily parsing web data in a distributive way.
## To install `bash pip install pifetcher `
## PYPI Link [https://pypi.org/project/pifetcher/]
## dependencies: - pyppeteer - BeautifulSoup4 - boto3
## features:
- event-callback-based interaction between user defined logic and the pre-disigned fetch worker
- process works in batches, library user will be able to capture the event of a batch of works have been finished
- easy to use, only needs to inherit the FetchWorker class and implement the basic call back functions
- it’s design to use message queue, enbles more than just one worker to perform data fetching in order to scale the application
## how to use:
Github: [https://github.com/gavinz0228/pifetcher]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size pifetcher-0.0.3.8.tar.gz (9.7 kB) | File type Source | Python version None | Upload date | Hashes View |