scrapy-item-ingest 0.1.1

Scrapy extension for database ingestion with job/spider tracking

Scrapy Item Ingest

A comprehensive Scrapy extension for ingesting scraped items, requests, and logs into PostgreSQL databases with advanced tracking capabilities. This library provides a clean, production-ready solution for storing and monitoring your Scrapy crawling operations with real-time data ingestion and comprehensive logging.

Documentation

Full documentation is available at: https://scrapy-item-ingest.readthedocs.io/en/latest/

Key Features

• 🔄 Real-time Data Ingestion: Store items, requests, and logs as they're processed
• 📊 Request Tracking: Track request response times, fingerprints, and parent-child relationships
• 🔍 Comprehensive Logging: Capture spider events, errors, and custom messages
• 🏗️ Flexible Schema: Support for both auto-creation and existing table modes
• ⚙️ Modular Design: Use individual components or the complete pipeline
• 🛡️ Production Ready: Handles both development and production scenarios
• 📝 JSONB Storage: Store complex item data as JSONB for flexible querying
• 🐳 Docker Support: Complete containerization with Docker and Kubernetes
• 📈 Performance Optimized: Connection pooling and batch processing
• 🔧 Easy Configuration: Environment-based configuration with validation
• 📊 Monitoring Ready: Built-in metrics and health checks

Installation

pip install scrapy-item-ingest

Development

Setting up for Development

git clone https://github.com/fawadss1/scrapy_item_ingest.git
cd scrapy_item_ingest
pip install -e ".[dev]"

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For support and questions:

• Email: [e-mail]
• Documentation: https://scrapy-item-ingest.readthedocs.io/
• Issues: Please report bugs and feature requests at GitHub Issues

Changelog

v0.1.1 (Current)

• Initial release
• Core pipeline functionality for items, requests, and logs
• PostgreSQL database integration with JSONB storage
• Comprehensive documentation and examples
• Production deployment guides
• Docker and Kubernetes support