Scrapy extension that logs errors to Sentry
Project description
city-scrapers-sentry
A Scrapy extension that logs spider errors to Sentry.
This project began as a fork of scrapy-sentry, which was developed by Jordi Llonch. We are grateful for his work and those of other contributors.
Note: while this extension captures errors from Scrapy spiders, it does not capture errors elsewhere in Scrapy's operation (e.g. the Scrapy pipeline).
Requirements:
Installation
pip install city-scrapers-sentry
Setup
Add SENTRY_DSN
and city_scrapers_sentry.extensions.Errors
extension to your Scrapy project's settings.py
file:
Example:
# sentry dsn
SENTRY_DSN = 'http://public:secret@example.com/1'
EXTENSIONS = {
"city_scrapers_sentry.extensions.Errors": 10,
}
Development
-
Install pipenv if you don't have it already.
-
At the project root, create a .env file and include the following. Replace
<your-sentry-dsn>
with your Sentry DSN:
SENTRY_DSN=<your-sentry-dsn>
- Run the following command from the project root to activate pipenv's virtual environment and install project dependencies:
pipenv sync --dev
When activated, pipenv should load your SENTRY_DSN env var from the .env file.
- Enter the example_project directory:
cd example_project
- Simulate a failed spider run. The spider will log an error to your Sentry account:
scrapy crawl example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for city-scrapers-sentry-1.0.0b1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25c0148fd53c4ea1d5f40b2d3059fe9cb90015418ac74836a177c89d6e3158ec |
|
MD5 | 0cd280038c6993a406d972e98120d87c |
|
BLAKE2b-256 | 12d6fabbd37e3ed6b890efc7ea8bdad638115a611913e45af643f8908a6c5506 |
Hashes for city_scrapers_sentry-1.0.0b1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a83a63e9b01a4f8acb9be0ff8da5199874b3d317e7482b02e6335b3222b0d1e1 |
|
MD5 | 750073abcb53b273af382a54209f7fb2 |
|
BLAKE2b-256 | d39e151dc48211019cdfa99a1ce6ebca02bd29d68ac340baccc98916ddda4704 |