jh-scrapyd 0.3.7

Preemptive scraping cluster

Jh Scrapyd

A preemptive scrapyd cluster built using Redis queues

• Free software: MIT license (including the work distributed under the Apache 2.0 licenWHEELse)
• Documentation: https://scrapyd.readthedocs.org/en/latest/

Installation

python >= 3.7.0

Install with pip from PyPI:

pip install jh_scrapyd

Configuration

editor scrapyd.conf

[api]
# interface md5 key
signature_key = 48jwmeX3iF6X0MdZy4atj2RpmNbQjAkm

# Interface signature expiration time
signature_expired_time = 300

[cluster]
# Add Redis to configure celarclear
host = 127.0.0.1
password = 
port = 6379
db = 1

# Is there a unified queue (mainly used for different projects to share queue priority)
is_unified_queue = 1

# Queue prefix
queue_prefix = jh_scrapyd