A Scrapy extension to be used, for example, with HTTP Callback Tasks of Celery project.
Installation
Install via pip:
pip install scrapy-notifications
Tested with
Python 2.7+
Scrapy 0.16
Setup
Basically you need configure SPIDER_NOTIFICATION_ENABLED, SPIDER_NOTIFICATION_URL and add the scrapy_notifications.extensions.SpiderNotification extension to your Scrapy project settings.py.
Example:
EXTENSIONS = {
'scrapy_notifications.extensions.SpiderNotification': 500,
}
SPIDER_NOTIFICATION_ENABLED = True
SPIDER_NOTIFICATION_URL = 'http://app.example.com/spider-task/'
By default HTTP requests are sent with the spider name attribute as an URL parameter. But optionally you can also specify a list of spider attributes passed as URL parameters on SPIDER_NOTIFICATION_ATTRS.:
SPIDER_NOTIFICATION_ATTRS = ['name', 'jobid']
Usage
For example crawling with the dmoz spider used in the Scrapy tutorial:
scrapy crawl -a jobid=fe648c57f8 dmoz
will generate a HTTP request like this:
http://app.example.com/spider-task/?name=dmoz&jobid=fe648c57f8
Contributing
Want to contribute? Great! Bug reports and code and documentation patches are greatly appretiated. Please file bugs and send pull requests using the issue tracker.
Metadata
Release files for scrapy-notifications 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapy-notifications-0.1.0.tar.gz | 2.3 kB | Without provenance | Details for scrapy-notifications-0.1.0.tar.gz |
Release files / scrapy-notifications-0.1.0.tar.gz
| Download URL | scrapy-notifications-0.1.0.tar.gz |
|---|---|
| Size | 2.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
17fd95ee4f66a66cd2a7c45fea1d51612f7f1b6c82b9c47f7e454cfd42189206
|
|
BLAKE2b-256 checksum How to use checksums |
5e90582f7443f106f2de57449f17d551171e2dce9b4cf869fe14827c939547cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for scrapy-notifications 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.