A Scrapy extension to be used, for example, with HTTP Callback Tasks of Celery project.
Installation
Install via pip:
pip install scrapy-notifications
Tested with
Python 2.7+
Scrapy 0.16
Setup
Basically you need configure SPIDER_NOTIFICATION_ENABLED, SPIDER_NOTIFICATION_URL and add the scrapy_notifications.extensions.SpiderNotification extension to your Scrapy project settings.py.
Example:
EXTENSIONS = {
'scrapy_notifications.extensions.SpiderNotification': 500,
}
SPIDER_NOTIFICATION_ENABLED = True
SPIDER_NOTIFICATION_URL = 'http://app.example.com/spider-task/'
By default HTTP requests are sent with the spider name attribute as an URL parameter. But optionally you can also specify a list of spider attributes passed as URL parameters on SPIDER_NOTIFICATION_ATTRS.:
SPIDER_NOTIFICATION_ATTRS = ['name', 'jobid']
Usage
For example crawling with the dmoz spider used in the Scrapy tutorial:
scrapy crawl -a jobid=fe648c57f8 dmoz
will generate a HTTP request like this:
http://app.example.com/spider-task/?name=dmoz&jobid=fe648c57f8
Contributing
Want to contribute? Great! Bug reports and code and documentation patches are greatly appretiated. Please file bugs and send pull requests using the issue tracker.
1 maintainer
Release files for scrapy-notifications 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-notifications-0.1.0.tar.gz | 2.3 kB | Details |
Release files / scrapy-notifications-0.1.0.tar.gz
| Download URL | scrapy-notifications-0.1.0.tar.gz |
|---|---|
| Size | 2.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
17fd95ee4f66a66cd2a7c45fea1d51612f7f1b6c82b9c47f7e454cfd42189206
|
|
BLAKE2b-256 checksum How to use checksums |
5e90582f7443f106f2de57449f17d551171e2dce9b4cf869fe14827c939547cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
