Scrapy extension that logs errors to Sentry
Project description
city-scrapers-sentry
A Scrapy extension that logs spider errors to Sentry.
This project began as a fork of scrapy-sentry, which was developed by Jordi Llonch. We are grateful for his work and those of other contributors.
Note: while this extension captures errors from Scrapy spiders, it does not capture errors elsewhere in Scrapy's operation (e.g. the Scrapy pipeline).
Requirements:
Installation
pip install city-scrapers-sentry
Setup
Add SENTRY_DSN
and city_scrapers_sentry.extensions.Errors
extension to your Scrapy project's settings.py
file:
Example:
# sentry dsn
SENTRY_DSN = 'http://public:secret@example.com/1'
EXTENSIONS = {
"city_scrapers_sentry.extensions.Errors": 10,
}
Development
-
Install pipenv if you don't have it already.
-
At the project root, create a .env file and include the following. Replace
<your-sentry-dsn>
with your Sentry DSN:
SENTRY_DSN=<your-sentry-dsn>
- Run the following command from the project root to activate pipenv's virtual environment and install project dependencies:
pipenv sync --dev
When activated, pipenv should load your SENTRY_DSN env var from the .env file.
- Enter the example_project directory:
cd example_project
- Simulate a failed spider run. The spider will log an error to your Sentry account:
scrapy crawl example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for city-scrapers-sentry-1.0.0a3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 000ed783665bd59e02a397bd45f60d1eb105fb24562f570df224a42e70c8244d |
|
MD5 | d19ba12d957f4d47ae3d109c37b25f14 |
|
BLAKE2b-256 | fd327072b2a84253bb5ecdc741b6ebd9d4b77da2278777573b18b2361ab5af2e |
Hashes for city_scrapers_sentry-1.0.0a3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e73a5638a0ddfd1550791f94ade9e10dedb4e5e0ef0c198f656cb25cf0778d8a |
|
MD5 | 6f17e0caba00b0614fc252a249226ad5 |
|
BLAKE2b-256 | c82b869bee13b4bdd51510d85a52f4c2ac24168cd3c23aa64f2d4740fecc7392 |