scrapyq

Scrapyq is designed to replace the SQLite backend by a Redis backend. In other words, all the queue management will be done using Redis.

Scrapyq is a fork of the original https://github.com/speakol-ads/scrapyd-redis implementation.

Install

pip install scrapyq

Config

To start using this library you just need to override the spiderqueue option in your scrapyd.conf file:

[scrapyd]
spiderqueue = scrapyq.spiderqueue.RedisSpiderQueue

If you want to customize the access to the database, you can add into your scrapyd.conf file:

[scrapyq]
queue_prefix = scrapyq.queue.
redis_db = 0
redis_host = localhost
redis_port = 6379
redis_username = 'admin'  # (Optional)
redis_password = 'password'  # (Optional)

Development and Testing

Installation

Install the package with development dependencies using the dev extra:

pip install -e ".[dev]"

Running tests

Testing requires a Redis instance. You can quickly start a Docker container using the provided Makefile command:

make redis-container

The standard test command runs both Ruff linting and the full pytest suite:

make test

Run code linting with Ruff:

make lint

Run documentation linting with mdformat:

make md-lint

Automatically format code and imports with Ruff:

make format

The coverage command runs linting, documentation linting, and generates a coverage report:

make coverage