APCloudy Pipeline

Scrapy integration that sends items, requests, logs, and stats to AP Cloudy with HMAC auth and full job log capture.

Install

pip install apcloudy-pipeline

Settings

APCLOUDY_URL = "https://your-api.com"
APCLOUDY_API_KEY = "your_public_api_key"
APCLOUDY_SECRET_KEY = "your_secret_key"
JOB_ID = 123
APCLOUDY_BATCH_SIZE = 50  # optional

ITEM_PIPELINES = {
    "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
}
DOWNLOADER_MIDDLEWARES = {
    "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
}
EXTENSIONS = {
    "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
    "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
    "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
}

Log level follows Scrapy LOG_LEVEL.

Payload

{
  "job_id": "123",
  "data": {
    "requests": [{ "url": "...", "status_code": 200, "success": true, "...": "..." }],
    "items": [{ "title": "...", "_ts": 1753358220 }],
    "logs": [{
      "index": 0,
      "time": "2026-07-01 13:02:55",
      "timestamp": 1751374975.12,
      "level": "INFO",
      "logger": "scrapy.core.engine",
      "message": "Spider opened",
      "exception": null
    }],
    "stats": { "item_scraped_count": 1, "finish_reason": "finished" }
  }
}

Logs cover startup → crawl → stats dump → closed (plus print() spider lines).

Auth

X-API-KEY / X-TIMESTAMP / X-SIGNATURE
POST {APCLOUDY_URL}/api/webhook/consume
signature = HMAC-SHA256(secret, timestamp + "." + body)

Notes

• Yield items from callbacks (not only return inside a generator).
• Data flushes on batch size, every 10s, and on engine/process stop.
• Requires Python 3.8+, Scrapy 2.0+, requests, w3lib, itemadapter.

License

MIT