This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

RabbitMQ Plug-in for Scrapy

Project Description
## A RabbitMQ Scheduler for Scrapy Framework.

Scrapy-rabbitmq-link is a tool that lets you feed URLs from RabbitMQ to Scrapy spiders, using the [Scrapy framework](

This is a modified version of scrapy-rabbitmq published by Royce Haynes in [GitHub](

## Installation

Using pip, type in your command-line prompt

pip install scrapy-rabbitmq-link

Or clone the repo and inside the scrapy-rabbitmq-link directory, type

python install

## Usage

### Step 1: In your scrapy settings, add the following config values:

# Enables scheduling storing requests queue in rabbitmq.
SCHEDULER = "scrapy_rabbitmq_link.scheduler.Scheduler"

# Don't cleanup rabbitmq queues, allows to pause/resume crawls.

# Schedule requests using a priority queue. (default)
SCHEDULER_QUEUE_CLASS = 'scrapy_rabbitmq_link.queue.SpiderQueue'

# Set expression for RabbitMQ URLs queue key
SCHEDULER_QUEUE_KEY = '%(spider)s'

# Provide host and port to RabbitMQ daemon
RABBITMQ_CONNECTION_PARAMETERS = 'amqp://user:pass@host:port/vhost'

# Response status to mark message as acknowledged and remove from queue

# Set middleware to status a successful remote procedure call
'scrapy_rabbitmq_link.middleware.RabbitMQMiddleware': 999


### Step 2: Add request building methods to Spider : _modify_request and _callback

#### Example:

from time import time
from scrapy.spiders import Spider

class CustomSpider(Spider):
""" Make requests using urls from RabbitMQ queue named same as spider

name = 'custom_spider'

# modify a request before firing. request already contains url received from RabbitMQ
def _modify_request(self, request):
request.meta['time'] = time()
return request

# callback to the response received
def _callback(self, response):
return self.parse(response)

def parse(self, response):
# extract stuff from response

### Step 3: Push URLs to RabbitMQ

#### Example:

#!/usr/bin/env python
import pika
import settings

connection = pika.BlockingConnection(pika.URLParameters(settings.RABBITMQ_CONNECTION_PARAMETERS))
channel =

# set queue name
queue = '%(spider)s' % {'spider':'custom'}

# publish links to queue
for url in open('urls.txt'):
url = url.strip(' \n\r')



### Step 4: Run spider using [scrapy client](

scrapy crawl custom_spider

## Contributing and Forking

See [Contributing Guidlines](CONTRIBUTING.MD)

## Releases

See the [changelog]( for release details.

| Version | Release Date |
| :-----: | :----------: |
| 0.1.0 | 2016-08-23 |

## Copyright & License

Copyright for portions of project "scrapy-rabbitmq" are held by Royce Haynes (c) 2014 as part of project "scrapy-rabbitmq-link". All other copyright for project "scrapy-rabbitmq-link" are held by Mantas Briliauskas (c) 2016.
Release History

Release History

This version
History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


History Node


Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
scrapy-rabbitmq-link-0.2.0.tar.gz (5.9 kB) Copy SHA256 Checksum SHA256 Source Oct 28, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting