Error Handling and Processing for your Scrapy Exceptions
Project description
scrapy-toolbox
A Python library that extends Scrapy with the following features:
- Error Saving to the Database Table "__errors" for manual error analysis (incl. traceback and response) and automated request reconstruction containing the following columns:
- failed_at
- spider
- traceback
- url (original url)
- request_method
- request_url
- request_meta (json dump that can be loaded with json.loads())
- request_cookies (json dump that can be loaded with json.loads())
- request_headers (json dump that can be loaded with json.loads())
- request_body
- response_status
- response_url
- response_headers (json dump that can be loaded with json.loads())
- response_body
- Error Processing with request reconstruction
- DatabasePipeline for SQLAlchemy
- Mapper to automaticaly map scrapy.Item on a database-object
- Mail Notification when an Exception occurs (HTTP Errors (404, 502, ...) are excluded and only stored in the Database)
- Automatic GitHub Issue creation when an Exception occurs (HTTP Errors (404, 502, ...) are excluded and only stored in the Database)
Requisites:
- Environment variable "PRODUCTION" for Produciton Mode for instance in your Dockerfile
- The ErrorSavingMiddleware defines an errback Callback for your Requests. If you want to make use of this Feature do not define any errback.
Installation
pip install --upgrade scrapy-toolbox
Setup
Add the scrapy_toolbox Middlewares to your Scrapy Project settings.py
and set your DATABASE_DEV and DATABASE.
# settings.py
SPIDER_MIDDLEWARES = {
'scrapy_toolbox.database.DatabasePipeline': 999,
'scrapy_toolbox.error_handling.ErrorSavingMiddleware': 1000,
'scrapy_toolbox.error_processing.ErrorProcessingMiddleware': 1000,
}
# Example when using a MySQL
DATABASE = {
'drivername': 'mysql+pymysql',
'username': '...',
'password': '...',
'database': '...',
'host': '...',
'port': '3306'
}
DATABASE_DEV = {
'drivername': 'mysql+pymysql',
'username': '...',
'password': '...',
'database': '...',
'host': '127.0.0.1',
'port': '3306'
}
CREATE_GITHUB_ISSUE = True # Toggle GitHub Issue creation
GITHUB_TOKEN = "..."
GITHUB_REPO = "janwendt/scrapy-toolbox" # for instance
SEND_MAILS = True # Toggle Mail Notification
MAIL_HOST = "..."
MAIL_FROM = "..."
MAIL_TO = "..."
Usage
Spider (Import ErrorCatcher first!!!):
from scrapy_toolbox.error_handling import ErrorCatcher
import scrapy
...
class XyzSpider(scrapy.Spider, metaclass=ErrorCatcher):
...
Database Pipeline:
# pipelines.py
from scrapy_toolbox.database import DatabasePipeline
import xy.items as items
import xy.model as model
class ScraperXYZPipeline(DatabasePipeline):
def __init__(self, settings):
super().__init__(settings, items, model)
# models.py
import scrapy_toolbox.database as db
# then use db.DeclarativeBase as your declarative base
class Car(db.DeclarativeBase):
...
Query Data:
# spiderXYZ.py
session = self.crawler.database_session
session.query(models.Market.id, models.Market.zip_code).all()
Process Errors:
scrapy crawl spider_xyz -a process_errors=True
Limitations
Syntax Errors in your settings.py are not handled.
Supported versions
This package works with Python 3. It has been tested with Scrapy up to version 1.4.0.
Tasklist
- [] add Database Error id to GitHub Issues
- [] check again fairgarage
Build Realease
python setup.py sdist bdist_wheel
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy-toolbox-0.3.0.tar.gz
(8.8 kB
view hashes)
Built Distribution
Close
Hashes for scrapy_toolbox-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 338b576f7dd4a70d069a1868f79484ca6cac4ec33ea4a382ee25ee71df266539 |
|
MD5 | 691ff0de2304d4b1f085c9764c46fd0f |
|
BLAKE2b-256 | c20429c79e94d29cf8eb356edb89e867ba577cb76866ed9c6d27b94c0d8344d6 |