acrawler-prometheus

The handler works with aCrawler and Prometheus.

Export statistics for:

• Concurrent requests
• Task (Requests, Items) counts
• Queue status

Installation

$ pip install acrawler_prometheus

Usage

Add Handler:

class MyCrawler(Crawler):
    middleware_config = {
        "acrawler_prometheus.PromExporter": 100,
    }

    config = {
        "PROMETHEUS_INTERVAL": 5
    }

Avaliable Config:

PROMETHEUS_ADDR = "localhost"
PROMETHEUS_PORT = 8000
PROMETHEUS_INTERVAL = 1  # exporting interval, in second