Skip to main content

Gerapy RabbitMQ

This is a package for supporting distribution in Scrapy using RabbitMQ, also this package is a module in Gerapy.

Installation

You can install with this command:

pip3 install gerapy-rabbitmq

Usage

Required configuration:

# Use RabbitMQ for queue
SCHEDULER = "gerapy_rabbitmq.scheduler.Scheduler"
SCHEDULER_QUEUE_KEY = '%(spider)s_requests'

# RabbitMQ Connection Parameters, see https://pika.readthedocs.io/en/stable/modules/parameters.html
RABBITMQ_CONNECTION_PARAMETERS = {
    'host': 'localhost'
}

# Use Redis for dupefilter
DUPEFILTER_CLASS = "gerapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_DUPEFILTER_KEY = '%(spider)s:dupefilter'

Optional configuration:

# RabbitMQ Queue Configuration
SCHEDULER_QUEUE_DURABLE = True
SCHEDULER_QUEUE_MAX_PRIORITY = 100
SCHEDULER_QUEUE_PRIORITY_OFFSET = 30
SCHEDULER_QUEUE_FORCE_FLUSH = True
SCHEDULER_PERSIST = False
SCHEDULER_IDLE_BEFORE_CLOSE = 0
SCHEDULER_FLUSH_ON_START = False
SCHEDULER_PRE_ENQUEUE_ALL_START_REQUESTS = True

More

For more detail, you can refer to example.

RabbitMQ Preview

Metadata

Release files for gerapy-rabbitmq 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gerapy-rabbitmq 0.1.1
File Size Uploaded Provenance Actions
gerapy-rabbitmq-0.1.1.tar.gz 6.3 kB Without provenance Details for gerapy-rabbitmq-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for gerapy-rabbitmq 0.1.1
File Interpreter ABI Platform Actions
gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl Python 2, Python 3 none any Details for gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl

Total release size: 13.0 kB

Release files / gerapy-rabbitmq-0.1.1.tar.gz

Download URL gerapy-rabbitmq-0.1.1.tar.gz
Size 6.3 kB
Tags Source
SHA-256 checksum
How to use checksums
1e944ee404c1ffe18c2f324a7438e2678311c1b496ac1f0da739711c1adea3f9
BLAKE2b-256 checksum
How to use checksums
7f05f2796ea580d466aab79f9836ad59a61d61e1ea152d9650b920f7ba12f789
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.13

Release files / gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl

Download URL gerapy_rabbitmq-0.1.1-py2.py3-none-any.whl
Size 6.7 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
b9a8b1131e5e48f2167c7282162e2f70c1e4576474eab7af81165075bb5a3663
BLAKE2b-256 checksum
How to use checksums
61ee7bc368c43e1eceff6a43fa11c89357c2dca12bf3d55ce216e7834daa6d04
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.13

Security for gerapy-rabbitmq 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page