Skip to main content

A Scrapy extension to be used, for example, with HTTP Callback Tasks of Celery project.

Installation

Install via pip:

pip install scrapy-notifications

Tested with

  • Python 2.7+

  • Scrapy 0.16

Setup

Basically you need configure SPIDER_NOTIFICATION_ENABLED, SPIDER_NOTIFICATION_URL and add the scrapy_notifications.extensions.SpiderNotification extension to your Scrapy project settings.py.

Example:

EXTENSIONS = {
    'scrapy_notifications.extensions.SpiderNotification': 500,
}
SPIDER_NOTIFICATION_ENABLED = True
SPIDER_NOTIFICATION_URL = 'http://app.example.com/spider-task/'

By default HTTP requests are sent with the spider name attribute as an URL parameter. But optionally you can also specify a list of spider attributes passed as URL parameters on SPIDER_NOTIFICATION_ATTRS.:

SPIDER_NOTIFICATION_ATTRS = ['name', 'jobid']

Usage

For example crawling with the dmoz spider used in the Scrapy tutorial:

scrapy crawl -a jobid=fe648c57f8 dmoz

will generate a HTTP request like this:

http://app.example.com/spider-task/?name=dmoz&jobid=fe648c57f8

Contributing

Want to contribute? Great! Bug reports and code and documentation patches are greatly appretiated. Please file bugs and send pull requests using the issue tracker.

Metadata

Release files for scrapy-notifications 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-notifications 0.1.0
File Size Uploaded Provenance Actions
scrapy-notifications-0.1.0.tar.gz 2.3 kB Without provenance Details for scrapy-notifications-0.1.0.tar.gz

Release files / scrapy-notifications-0.1.0.tar.gz

Download URL scrapy-notifications-0.1.0.tar.gz
Size 2.3 kB
Tags Source
SHA-256 checksum
How to use checksums
17fd95ee4f66a66cd2a7c45fea1d51612f7f1b6c82b9c47f7e454cfd42189206
BLAKE2b-256 checksum
How to use checksums
5e90582f7443f106f2de57449f17d551171e2dce9b4cf869fe14827c939547cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for scrapy-notifications 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page