APCloudy Pipeline

A Scrapy integration that sends items, requests, logs, and spider statistics to your AP Cloudy backend using unified batch processing and HMAC-SHA256 authentication.

---

✨ Features

• 📦 Unified Data Sending - All data (items, requests, logs, stats) sent in a single API call
• 🚀 Batch Processing - Automatic batching every 10 items or on spider close
• 🌐 Complete Request Tracking - Logs both successful and failed requests with detailed error information
• 📊 Spider Statistics - Comprehensive spider performance metrics
• 🧾 Log Forwarding - Captures spider, user, and Scrapy internal logs
• 🔐 Secure Authentication - HMAC-SHA256 signature-based API communication
• ⚡ High Performance - Thread-safe data collection with minimal overhead
• 🎯 Zero Configuration - Works out of the box with sensible defaults

---

📦 Installation

pip install apcloudy-pipeline

---

⚙️ Configuration

Add these settings to your Scrapy settings.py (or spider custom_settings):

# Required: API credentials
APCLOUDY_URL = "https://your-api.com"  # Base URL (webhook path added automatically)
APCLOUDY_API_KEY = "your_public_api_key"
APCLOUDY_SECRET_KEY = "your_secret_key"
JOB_ID = 123  # Can also be passed via spider args: -a JOB_ID=456

# Optional: Batch size (default: 50)
APCLOUDY_BATCH_SIZE = 50

# Required: Item Pipeline
ITEM_PIPELINES = {
    "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
}

# Required: Error Middleware (failed requests)
DOWNLOADER_MIDDLEWARES = {
    "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
}

# Required: Extensions (requests, logs, stats)
EXTENSIONS = {
    "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
    "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
    "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
}

---

🏗️ Architecture

Data Flow

Spider execution
       │
       ├── Requests ──┐
       ├── Items ─────┼──► DataCollector (thread-safe)
       ├── Logs ──────┤           │
       └── Stats ─────┘           ▼
                          Batch trigger?
                          • size >= APCLOUDY_BATCH_SIZE
                          • every 10 seconds
                          • spider closes
                                  │
                                  ▼
                          APCloudyClient (HMAC)
                                  │
                                  ▼
                     POST /api/webhook/consume

---

📡 API Payload Structure

All data is sent in a single unified payload:

{
  "job_id": "123",
  "data": {
    "requests": [
      {
        "url": "https://example.com/product",
        "method": "GET",
        "status_code": 200,
        "response_time": 1.23,
        "fingerprint": "f3045685b89f920b3faefc7d3df2d3c88bdab393",
        "error": null,
        "success": true
      }
    ],
    "items": [
      {
        "title": "Product Name",
        "price": "99.99",
        "url": "https://example.com/product",
        "_ts": 1753358220
      }
    ],
    "logs": [
      {
        "level": "INFO",
        "message": "Spider started",
        "exception": null
      }
    ],
    "stats": {
      "item_scraped_count": 1,
      "response_received_count": 1,
      "finish_time": "2026-07-24T13:08:29.571736+00:00",
      "finish_reason": "finished"
    }
  }
}

Each scraped item includes _ts — the Unix timestamp (seconds) when the pipeline collected it.

---

Components

Component	Role
APCloudyItemPipeline	Collects items (adds _ts), batches and sends data
APCloudyRequestLogger	Logs successful HTTP responses
APCloudyErrorMiddleware	Logs failed requests / exceptions
APCloudyLoggingExtension	Forwards Python/Scrapy logs
APCloudyStatsExtension	Collects spider stats on close
DataCollector	Thread-safe shared buffer
APCloudyClient	HMAC-signed HTTP client

---

Authentication

X-API-KEY: {your_public_key}
X-TIMESTAMP: {unix_timestamp}
X-SIGNATURE: {hmac_sha256(secret_key, timestamp + "." + json_body)}
Content-Type: application/json

message = f"{timestamp}.{json_body}"
signature = HMAC_SHA256(secret_key, message)

---

Requirements

• Python 3.8+
• Scrapy 2.0+
• requests, w3lib, itemadapter

---

Advanced Configuration

Batch size

APCLOUDY_BATCH_SIZE = 50   # default
APCLOUDY_BATCH_SIZE = 1    # send as soon as possible

Data is also flushed every 10 seconds and always on spider close.

Job ID via spider args

scrapy crawl myspider -a JOB_ID=456

Backend endpoint

POST {APCLOUDY_URL}/api/webhook/consume

Example: https://your-api.com → https://your-api.com/api/webhook/consume

---

Example Spider

Important: use yield for items. If the callback also yields Requests, a trailing return item is ignored by Python/Scrapy.

import scrapy


class MySpider(scrapy.Spider):
    name = "myspider"

    custom_settings = {
        "APCLOUDY_URL": "https://your-api.com",
        "APCLOUDY_API_KEY": "your_public_api_key",
        "APCLOUDY_SECRET_KEY": "your_secret_key",
        "JOB_ID": 123,
        "ITEM_PIPELINES": {
            "apcloudy_pipeline.pipelines.APCloudyItemPipeline": 300,
        },
        "DOWNLOADER_MIDDLEWARES": {
            "apcloudy_pipeline.middleware.APCloudyErrorMiddleware": 50,
        },
        "EXTENSIONS": {
            "apcloudy_pipeline.request_logger.APCloudyRequestLogger": 100,
            "apcloudy_pipeline.extensions.APCloudyLoggingExtension": 100,
            "apcloudy_pipeline.extensions.APCloudyStatsExtension": 100,
        },
    }

    def start_requests(self):
        for url in ["https://example.com/page1", "https://example.com/page2"]:
            yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "price": response.css(".price::text").get(),
            "url": response.url,
        }

No extra spider logic is required for AP Cloudy — enable the pipeline/extensions and yield items.

---

Troubleshooting

Items are always empty / item_scraped_count missing

1. Confirm the callback yields the item (not only return inside a generator).
2. Confirm APCloudyItemPipeline is in ITEM_PIPELINES.
3. Check spider logs for Failed to send APCloudy batch.

Data not being sent

1. Verify APCLOUDY_URL, APCLOUDY_API_KEY, APCLOUDY_SECRET_KEY, and JOB_ID.
2. Ensure pipeline, middleware, and extensions are enabled.
3. Check network access to {APCLOUDY_URL}/api/webhook/consume.

Failed requests not logged

1. Add APCloudyErrorMiddleware to DOWNLOADER_MIDDLEWARES.
2. Use a low priority (e.g. 50) so it sees exceptions early.

Stats missing

1. Enable APCloudyStatsExtension.
2. Stats are attached on spider close.

---

Changelog

0.1.7

• Add _ts (Unix timestamp) to every item before send
• Use ItemAdapter for dict / Item / dataclass / attrs items
• Safer JSON serialization for item fields (datetime, Decimal, etc.)
• Requeue batches on send failure instead of silently dropping them
• Periodic flush every 10 seconds
• Docs: correct default batch size (50), yield guidance, troubleshooting

0.1.6

• Previous stable release

---

🤝 Contributing

MIT

---

📧 Support

Pull requests are welcome.

---

Support

Open an issue on GitHub.