apcloudy-pipeline 0.1.0

Scrapy pipeline & extensions for AP Cloudy (logs, stats, requests, items)

apcloudy-pipeline

apcloudy-pipeline is a Scrapy integration that sends items, requests, logs, and spider statistics to the Your backend using secure HMAC-based authentication.

---

Features

• 📦 Item forwarding using Scrapy Item Pipeline
• 🌐 Request and response logging
• 📊 Spider statistics reporting
• 🧾 Spider, user, and Scrapy internal log forwarding
• 🔐 HMAC-secured API communication

---

Installation

pip install apcloudy-pipeline

Configuration

Add the following settings to your Scrapy project's settings.py file:

APCLOUDY_API_URL = "http://localhost:8000/api/v1/"
APCLOUDY_API_KEY = "api_test_1234567890"
APCLOUDY_SECRET_KEY = "secret_test_1234567890"
JOB_ID = 123

Item Pipeline (Required)

The item pipeline is required to send scraped items to the backend.

ITEM_PIPELINES = {
    "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
}

Extensions (Optional)

Enable the following extensions if you want to send requests, logs, and spider statistics.

EXTENSIONS = {
    "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 400,
    "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 510,
    "apcloudy_pipeline.extensions.APCloudyStatsExtension": 520,
}

Extensions Overview

• APCloudyRequestLogger Captures request and response metadata such as URL, HTTP method, status code, timing, and fingerprint.

• APCloudyLoggingExtension Sends spider logs, user logs, Scrapy internal logs, and exception tracebacks to the backend.

• APCloudyStatsExtension Sends final spider statistics when the crawl finishes.