Skip to main content

A series distributed components for Scrapy framework

Project description


Scrapy-Distributed is a series of components for you to develop a distributed crawler base on Scrapy in an easy way.

Now! Scrapy-Distributed has supported RabbitMQ Scheduler and RedisBloom DupeFilter. You can use either of those in your Scrapy's project very easily.

AMQP(RabbitMQ), Bloom Filter on Redis (RedisBloom) for Scrapy


  • RabbitMQ Scheduler
    • Support custom declare a RabbitMQ's Queue. Such as passive, durable, exclusive, auto_delete, and all other options.
  • RedisBloom DupeFilter
    • Support custom the key, errorRate, capacity, expansion and auto-scaling(noScale) of a bloom filter.


  • RabbitMQ Item Pipeline
  • RabbitMQ Scheduler


  • Python >= 3.6
  • Scrapy >= 1.8.0
  • Pika >= 1.0.0
  • RedisBloom >= 0.2.0
  • Redis >= 3.0.1


There is a simple demo in examples/simple_example. Here is the fast way to use Scrapy-Distributed.

Step 0:

pip install scrapy-distributed


git clone && cd scrapy-distributed
&& python install

If you don't have the required environment for tests:

# pull and run a RabbitMQ container.
docker run 
# pull and run a RedisBloom container.
docker run 

Step 1:

Only by change SCHEDULER, DUPEFILTER_CLASS and add some configs, you can get a distributed crawler in a moment.

SCHEDULER = "scrapy_distributed.schedulers.DistributedScheduler"
SCHEDULER_QUEUE_CLASS = "scrapy_distributed.queues.amqp.RabbitQueue"
RABBITMQ_CONNECTION_PARAMETERS = "amqp://guest:guest@localhost:5672/example/?heartbeat=0"
DUPEFILTER_CLASS = "scrapy_distributed.dupefilters.redis_bloom.RedisBloomDupeFilter"
BLOOM_DUPEFILTER_REDIS_URL = "redis://:@localhost:6379/0"
    "redis_cls": "redisbloom.client.Client"

Step 2:

scrapy crawl <your_spider>

Reference Project



Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Scrapy-Distributed, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size Scrapy-Distributed-0.0.3.tar.gz (11.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page