Scrapy Ingest

A Scrapy addon that saves items, requests, logs, and stats to PostgreSQL or MySQL — with parent_url tracking, failed-request errors, and full job log capture (including print()).

Install

pip install scrapy-ingest

Minimal setup (settings.py)

Only the item pipeline is required — requests, logs, stats, parent_url, and error logging are enabled automatically:

ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.DbInsertPipeline": 300,
}

# PostgreSQL
DB_URL = "postgresql://user:password@localhost:5432/database"
# MySQL
# DB_URL = "mysql://user:password@localhost:3306/database"
# Or discrete fields (avoids URL encoding):
# DB_TYPE = "postgres"   # or "mysql"
# DB_HOST = "localhost"
# DB_PORT = 5432         # MySQL: 3306
# DB_USER = "user"
# DB_PASSWORD = "password"
# DB_NAME = "database"

# Optional
CREATE_TABLES = True     # auto-create tables on first run (default True)
JOB_ID = 1               # or omit; a unique id is generated per crawl
INGEST_BATCH_SIZE = 50   # flush when this many rows are buffered

Run your spider:

scrapy crawl your_spider

Log level follows Scrapy LOG_LEVEL.

What is stored

Request parent_url is the page that scheduled the request (e.g. sitemap → product). Start URLs are null.

Data flushes on batch size, every 10s, and on engine/process stop.

When the spider closes, a crawl summary is printed (job, database, tables, items, requests, logs, errors, elapsed time) even if LOG_LEVEL is ERROR. Set INGEST_SHOW_SUMMARY = False to hide it.

Troubleshooting

• Password has special characters like @ or $?
  • In a URL, encode them: @ -> %40, $ -> %24.
  • Example: postgresql://user:PAK%40swat1%24@localhost:5432/db
  • Or use the discrete fields (no encoding needed).
• Yield items from callbacks (not only return inside a generator).

Useful settings (optional)

• DB_TYPE (default: postgres) — postgres / postgresql or mysql / mariadb when building a URL from DB_* fields
• INGEST_BATCH_SIZE (default: 50) — flush when this many items+requests+logs are buffered
• INGEST_FLUSH_INTERVAL (default: 10) — periodic flush in seconds
• INGEST_SHOW_SUMMARY (default: True) — print crawl summary tables when the spider closes
• CREATE_TABLES (default: True) — create tables on startup
• ITEMS_TABLE, REQUESTS_TABLE, LOGS_TABLE, JOBS_TABLE — override table names
• TIMEZONE (default: Asia/Karachi) — timezone for created_at
• JOB_ID — omit to auto-generate a unique id (Rs_Spider-178826754-a1b2c3)

Standalone components

If you only want part of the collection:

# Items only
ITEM_PIPELINES = {"scrapy_ingest.pipelines.ItemsPipeline": 300}

# Requests only (parent_url + errors)
ITEM_PIPELINES = {"scrapy_ingest.pipelines.RequestsPipeline": 300}

# Logs only
EXTENSIONS = {"scrapy_ingest.extensions.LoggingExtension": 500}

Links

• Docs: https://scrapy-ingest.readthedocs.io/
• Changelog: docs/development/changelog.rst
• Issues: https://github.com/fawadss1/scrapy_item_ingest/issues

License

MIT License. See LICENSE.