Redis-based scheduler and Message queue Spider for Scrapy, Provide more flexible and practical ways for Scrapy
Project description
# SoDo
## Overview
Redis-based scheduler and Message queue Spider for Scrapy, Provide more flexible and practical ways for Scrapy.
For more information [Document](https://ycvbcvfu.github.io/SoDo) [中文文档](https://ycvbcvfu.github.io/SoDo/#/zh-cn/):
## Requirements
Python 3
Scrapy >=1.6
## Features
Distributed
> Redis-based scheduler
Hooks feature
> new feature for Scrapy projects
Flexible parameter configuration
> Kafka , Redis client parameter set by yourself.
Filter interface
> Built-in BloomFiler and OriginalFilter
## Installation
`python pip install -U sodo -i https://pypi.org/simple `
## Contribution guidelines
If you want to contribute to SoDo, please see [CONTRIBUTING Doc](https://github.com/ycvbcvfu/SoDo/blob/master/CONTRIBUTING.md)
## For more information
[HelloWorld Tutorial](https://github.com/ycvbcvfu/SoDo/blob/master/examples/helloworld)
![wechat](docs/picture/wechat.JPG)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.