scrapyd_UI 0.9.9

Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization

ScrapydWeb: Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization
==========================

[](https://pypi.org/project/scrapydweb/)
[](https://pypi.org/project/scrapydweb/)
[](https://coveralls.io/github/my8100/scrapydweb?branch=master)
[](https://github.com/my8100/scrapydweb/blob/master/LICENSE)
[](https://twitter.com/intent/tweet?text=ScrapydWeb:%20Full-featured%20web%20UI%20for%20Scrapyd%20cluster%20management,%20Scrapy%20log%20analysis%20%26%20visualization%20%23python%20%20%23scrapy%20%23scrapyd%20%23webscraping%20%23scrapydweb%20@my8100_%20&url=https%3A%2F%2Fgithub.com%2Fmy8100%2Fscrapydweb)

Features
---------------

- Scrapyd Cluster Management
- Group, filter and select any numbers of nodes
- Execute command on multinodes with one click

- Scrapy Log Analysis
- Stats collection
- Progress visualization
- Logs categorization

- All Scrapyd API Supported
- Deploy project, Run Spider, Stop job
- List projects/versions/spiders/running_jobs
- Delete version/project

- Others
- Auto eggifying
- Basic auth for web UI
- Accessing Scrapyd servers protected by basic auth


Devolopers
---------------
- [my8100](https://github.com/my8100)
- [simplety](https://github.com/simplety) (Front-End)


Installation
------------

To install ScrapydWeb, simply use pip:

```
pip install scrapydweb
```


Start Up
------------

1. Run `scrapydweb -h` to get help,
and a config file named **scrapydweb_settings_vN.py** (N for a number) would be copied to current working directory,
then you can custom settings in it.
2. Run `scrapydweb`
3. Visit [http://127.0.0.1:5000](http://127.0.0.1:5000) **(It's recommended to use Google Chrome to get better experience.)**


Settings
---------------
[default_settings.py](https://github.com/my8100/scrapydweb/blob/master/scrapydweb/default_settings.py)


Screenshots
------------

- Overview

- Dashboard

- Log Analysis
- Stats collection

- Progress visualization

- Logs categorization

- Deploy a Project

- Run a Spider

- Manage Projects