Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization
Project description
ScrapydWeb: Full-featured web UI for Scrapyd cluster management, Scrapy log analysis & visualization
Features
-
Scrapyd Cluster Management
- Group, filter and select any numbers of nodes
- Execute command on multinodes with one click
-
Scrapy Log Analysis
- Stats collection
- Progress visualization
- Logs categorization
-
All Scrapyd API Supported
- Deploy project, Run Spider, Stop job
- List projects/versions/spiders/running_jobs
- Delete version/project
-
Enhancements
- Basic auth for web UI
- HTML caching for the Log and Stats page
- Auto eggify your projects
- Email notice with multi-triggers provided
Devolopers
Installation
To install ScrapydWeb, simply use pip:
pip install scrapydweb
Start Up
- Run
scrapydweb -h
to get help, then a config file named scrapydweb_settings_vN.py (N for a number) would be copied to current working directory, in which you can custom settings. - Run
scrapydweb
- Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)
Settings
Screenshots
-
Overview
-
Dashboard
-
Log Analysis
-
stats collection
-
progress visualization
-
logs categorization
-
-
Deploy Project
-
Run Spider
-
Manage Projects
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapydweb-1.0.0rc1.tar.gz
(642.9 kB
view hashes)
Built Distribution
scrapydweb-1.0.0rc1-py3-none-any.whl
(680.3 kB
view hashes)
Close
Hashes for scrapydweb-1.0.0rc1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a448162550957e7ea64ca495e6d6567b8cd7d7fb83c29ba15b8ea798b20d0ddb |
|
MD5 | d3cf2e864bdfad1863411903289256ad |
|
BLAKE2b-256 | b39e528eab62395f8a36283aaa77fea95b88fcb25f440ee9dca1b9425b702a72 |