Skip to main content

Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported

Project description

ScrapydWeb: Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported

Features

  • Multinode Scrapyd Servers

    • Group, filter and select any numbers of nodes
    • Execute command on multinodes with one click
  • Scrapy Log Analysis

    • Stats collection
    • Progress visualization
    • Logs categorization
  • All Scrapyd API Supported

    • Deploy project, Run Spider, Stop job
    • List projects/versions/spiders/running_jobs
    • Delete version/project
  • Others

    • Basic auth for web UI

Maintainer

Installation

To install ScrapydWeb, simply use pip:

$ pip install scrapydweb

Start Up

Run "scrapydweb -h" to get help, and a config file named "scrapydweb_settings.py" would be copied to the working directory, then you can custom config with it.

$ scrapydweb

Visit http://127.0.0.1:5000

Screenshot

  • Overview overview

  • Dashboard dashboard

  • Log Analysis

    • Stats collection log_stats

    • Progress visualization log_chart

    • Logs categorization log_extracted

  • Deploy Project deploy

  • Run Spider run

  • Manage Projects manage

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapydweb-0.9.4.tar.gz (575.0 kB view hashes)

Uploaded source

Built Distribution

scrapydweb-0.9.4-py3-none-any.whl (598.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page