This is test repo of MonAnt-Crawler-Parser
Crawler using Scrapy with Celery integration
Requirements
These libraries are needed to be installed
- Scrapy (pip install scrapy, alternative for Windows: conda install -c conda-forge scrapy)
- rabbitmq-server
- sudo apt-get install rabbitmq-server
- brew install rabbitmq
- Celery (pip install celery)
- Twisted with newer version (pip3 install git+https://github.com/twisted/twisted.git@trunk)
- Redis
FlaskApi application must be running and provide API.
If needed, set environment variables MONANT_WEB_MONITORING_CENTRAL_STORAGE_USERNAME and MONANT_WEB_MONITORING_CENTRAL_STORAGE_PASSWORD depending on what API client stored in central data storage database should be used to authorize web monitoring application.
Running
- Rabbitmq-server must be started (using command {RABBIT_PATH}/sbin/rabbitmq-server) \
- Start celery worker using command (inside folder monant-crawler-parser): celery -A celeryapp.application worker --loglevel=info
- Run worker.py
1 maintainer
Release files for tst-monant 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tst_monant-0.1.3.tar.gz | 2.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tst_monant-0.1.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 6.0 kB
Release files / tst_monant-0.1.3.tar.gz
| Download URL | tst_monant-0.1.3.tar.gz |
|---|---|
| Size | 2.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
70b82bee10e51216b018268adac7045b76f056ef90b14bb01bf4ebc2cc5dee1d
|
|
BLAKE2b-256 checksum How to use checksums |
ca825fd5755a481ba28ee370506bcae2e890e6da217fe625ddf4803179bccef7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.9.15
|
Release files / tst_monant-0.1.3-py3-none-any.whl
| Download URL | tst_monant-0.1.3-py3-none-any.whl |
|---|---|
| Size | 3.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b2a2143bbe7e9322a8bea7b2622f280cc0bb0754339825e4c3a433c2e9514620
|
|
BLAKE2b-256 checksum How to use checksums |
42c6445e61640e02a33b2ec69fc0ebe766e7b0b6da0928e662496b2056477c86
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.9.15
|
