Gerapy RabbitMQ
This is a package for supporting distribution in Scrapy using RabbitMQ, also this package is a module in Gerapy.
Installation
You can install with this command:
pip3 install gerapy-rabbitmq
Usage
Required configuration:
# Use RabbitMQ for queue
SCHEDULER = "gerapy_rabbitmq.scheduler.Scheduler"
SCHEDULER_QUEUE_KEY = '%(spider)s_requests'
# RabbitMQ Connection Parameters, see https://pika.readthedocs.io/en/stable/modules/parameters.html
RABBITMQ_CONNECTION_PARAMETERS = {
'host': 'localhost'
}
# Use Redis for dupefilter
DUPEFILTER_CLASS = "gerapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_DUPEFILTER_KEY = '%(spider)s:dupefilter'
Optional configuration:
# RabbitMQ Queue Configuration
SCHEDULER_QUEUE_DURABLE = True
SCHEDULER_QUEUE_MAX_PRIORITY = 100
SCHEDULER_QUEUE_PRIORITY_OFFSET = 30
SCHEDULER_QUEUE_FORCE_FLUSH = True
SCHEDULER_PERSIST = False
SCHEDULER_IDLE_BEFORE_CLOSE = 0
SCHEDULER_FLUSH_ON_START = False
SCHEDULER_PRE_ENQUEUE_ALL_START_REQUESTS = True
More
For more detail, you can refer to example.
RabbitMQ Preview
Metadata
Release files for gerapy-rabbitmq 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gerapy-rabbitmq-0.1.1.tar.gz | 6.3 kB | Without provenance | Details for gerapy-rabbitmq-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details for gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl |
Total release size: 13.0 kB
Release files / gerapy-rabbitmq-0.1.1.tar.gz
| Download URL | gerapy-rabbitmq-0.1.1.tar.gz |
|---|---|
| Size | 6.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1e944ee404c1ffe18c2f324a7438e2678311c1b496ac1f0da739711c1adea3f9
|
|
BLAKE2b-256 checksum How to use checksums |
7f05f2796ea580d466aab79f9836ad59a61d61e1ea152d9650b920f7ba12f789
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
Release files / gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl
| Download URL | gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl |
|---|---|
| Size | 6.7 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
b9a8b1131e5e48f2167c7282162e2f70c1e4576474eab7af81165075bb5a3663
|
|
BLAKE2b-256 checksum How to use checksums |
61ee7bc368c43e1eceff6a43fa11c89357c2dca12bf3d55ce216e7834daa6d04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.13
|
Security for gerapy-rabbitmq 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.