scrapy-delta-guard 1.0.0

A Scrapy extension to detect data changes (deltas) between scraped items and a database.

🛡️ Scrapy DeltaGuard

Scrapy DeltaGuard is a powerful and easy-to-integrate Scrapy extension that monitors changes (deltas) between the data you scrape and the data already present in your database.

It helps maintain data integrity by monitoring data drift, reduces noisy or incorrect updates, and can automatically trigger alerts (e.g., to Slack or Jira) or stop a spider when significant changes occur.

🚀 Why use DeltaGuard?

Large-scale web scraping projects commonly face issues that corrupt data quality:

• Silent layout changes on target websites that break selectors.
• Inconsistent formatting for data like phone numbers (123-456 vs 123456).
• Transient incorrect data appearing in sponsored or related content blocks.

DeltaGuard helps you:

• Detect real content changes, not just formatting noise.
• Avoid cascading bad data writes by automatically halting crawls if too many fields drift.
• Integrate seamlessly without rewriting your existing item pipeline logic.
• Notify downstream systems (Jira, Slack, etc.) when data quality issues are found.

---

Installation

pip install scrapy-delta-guard

---

⚙️ Quick Start Guide

Follow these three steps to get DeltaGuard running in your project.

1. Configure Your Database Session for Detached Object Handling

To avoid SQLAlchemy’s DetachedInstanceError during delta checking, configure your SQLAlchemy session with:

from sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=engine, expire_on_commit=False)

This retains loaded database objects on commit, allowing access during Scrapy’s async pipeline.

2. Configure settings.py

Enable the extension and define field monitoring with flexible thresholds and options.

EXTENSIONS = {
    'deltaguard.extension.DeltaGuard': 500,
}

DELTA_GUARD_ENABLED = True

DELTA_GUARD_BATCH_SIZE = 50

DELTA_GUARD_DEFAULT_THRESHOLD = '5%'

DELTA_GUARD_FIELDS_CONFIG = [
    {'name': 'email'},  # simple shorthand for same db/spider field
    {'name': 'phone_number', 'threshold': 10},  # 10% threshold
    {
        'name': 'years_experience',
        'db_var': 'years_exp',  # different db attribute
        'spider_var': 'years_exp_spider',  # different spider field
        'threshold': '15%'
    },
]

DELTA_GUARD_DB_NONE_IS_DELTA = True
DELTA_GUARD_SPIDER_NONE_IS_DELTA = False

DELTA_GUARD_STOP_SPIDER_ON_HIGH_DELTA = True

DELTA_GUARD_JIRA_FUNC = 'my_project.utils.create_jira_ticket'

DELTA_GUARD_SLACK_WEBHOOK = 'https://hooks.slack.com/services/your/webhook/url'

LOG_LEVEL = 'DEBUG'

3. Update Your Scrapy Item

Ensure your Scrapy Item class includes the db_item field to avoid KeyError:

import scrapy

class YourItem(scrapy.Item):
    # ... your existing fields ...
    db_item = scrapy.Field()

4. Attach DB Items Using the Adapter in Your Pipelines

from deltaguard.adapter import DeltaGuardAdapter

class YourPipeline:
    def process_item(self, item, spider):
        db_item = self.session.query(YourModel).filter_by(email=item.get('email')).first()
        DeltaGuardAdapter.attach(item, db_item)
        return item

---

How Does DeltaGuard Work?

• The extension compares the fields in DELTA_GUARD_FIELDS_CONFIG between the scraped item and its corresponding database record.
• Differences are accumulated in batch sized groups defined by DELTA_GUARD_BATCH_SIZE.
• If deltas for any specific field exceed their configured percentage threshold during a batch, alerts are sent.
• Optionally, the spider is stopped immediately to prevent cascading bad data writes.

---

Configuration Reference

Setting	Type	Default	Description
DELTA_GUARD_ENABLED	bool	False	Enables or disables the extension globally.
DELTA_GUARD_FIELDS_CONFIG	list[dict]	[]	Fields to monitor with optional threshold, db_var, and spider_var.
DELTA_GUARD_BATCH_SIZE	int	50	Number of items processed per batch evaluation.
DELTA_GUARD_DEFAULT_THRESHOLD	str or float	5%	Default batch delta threshold (percentage) if none specified per field.
DELTA_GUARD_DB_NONE_IS_DELTA	bool	False	Treats a None in DB as delta if spider has a value.
DELTA_GUARD_SPIDER_NONE_IS_DELTA	bool	False	Treats a None in spider as delta if DB has a value.
DELTA_GUARD_STOP_SPIDER_ON_HIGH_DELTA	bool	True	Stops the spider when any field delta threshold is exceeded.
DELTA_GUARD_JIRA_FUNC	str	None	Dotted path to alert function (e.g., JIRA ticket creator).
DELTA_GUARD_SLACK_WEBHOOK	str	None	Slack Incoming Webhook URL for notifications.

---

Advanced Field Configuration

The DELTA_GUARD_FIELDS_CONFIG allows flexible definitions.

DELTA_GUARD_FIELDS_CONFIG = [
    {'name': 'email'},                 # Simple shorthand
    {'name': 'phone_number', 'db_var': 'phone', 'spider_var': 'contact_phone'},  # Custom fields
    {'name': 'salary', 'threshold': 15},  # 15% threshold as integer
    {'name': 'location', 'threshold': '25%'},  # 25% threshold as string
]

---

Using safe_commit to Prevent Data Corruption

If we enable DELTA_GUARD_STOP_SPIDER_ON_HIGH_DELTA it will gracefully stop the spider , however it will process the requests in queue and may write a few more batches of data. To prevent it we can use the safe_commit utility.

The safe_commit utility ensures you only commit your SQLAlchemy session if DeltaGuard has not flagged a high-delta event. If the flag was set, it automatically rolls back the session to prevent potentially bad/partial data saves.

Example usage in your pipeline:

from deltaguard.adapter import safe_commit

class YourDatabasePipeline:
def close_spider(self, spider):
# This checks the flag and decides to commit or rollback
safe_commit(self.session, spider)
self.session.close()

If you ever need to force a commit even when a high delta was detected:

safe_commit(self.session, spider, force_commit=True)

• The function returns True if committed, False if rolled back.

This wrapper lets you centralize all your session commit logic and align with how DeltaGuard protects your database from corrupt/incomplete batches.

CSV Delta Logs on Slack (Optional)

DeltaGuard can automatically generate and send detailed CSV logs to Slack when high delta thresholds are exceeded. This gives you a complete audit trail of what changed.

Configuration

In your settings.py

Enable CSV logs on Slack
DELTA_GUARD_LOGS_ON_SLACK = True

Required: Slack Bot Token (not webhook)
DELTA_GUARD_SLACK_BOT_TOKEN = "xoxb-your-bot-token-here"

Required: Slack Channel ID (not channel name)
Right Click on Channel >> View Channel Details
DELTA_GUARD_SLACK_CHANNEL_ID = "C01234ABCDE"

Setting up Slack Bot for File Uploads

To enable CSV file uploads, you need a Slack Bot Token (webhooks don't support file uploads):

1. Create a Slack App:

   • Go to https://api.slack.com/apps
   • Click "Create New App" → "From scratch"
   • Give it a name (e.g., "DeltaGuard Bot")

2. Add Bot Permissions:

   • Navigate to "OAuth & Permissions"
   • Under "Bot Token Scopes", add:
     • files:write (to upload files)
     • chat:write (to post messages)

3. Install the App:

   • Click "Install to Workspace"
   • Copy the "Bot User OAuth Token" (starts with xoxb-)

4. Get Your Channel ID:

   • In Slack, right-click your channel name
   • Select "View channel details"
   • Scroll to bottom and copy the Channel ID

5. Invite Bot to Channel:

   • In your Slack channel, type: /invite @DeltaGuard Bot

CSV Format

The generated CSV includes the following columns (sorted by field name):

• db_item_id: Primary key of the database record
• field: Field name that changed
• old_value: Value in the database
• new_value: Value from the spider

Note: Only deltas from fields that exceeded the threshold are included in the CSV.

Example Output

When a high delta is detected, you'll receive:

1. A text alert showing which fields exceeded thresholds
2. A CSV file attachment with detailed change logs

The CSV filename format: deltaguard_{spider_name}_deltas.csv

License

MIT License