Skip to main content

Redis-based scheduler and Message queue Spider for Scrapy, Provide more flexible and practical ways for Scrapy

Project description

# SoDo

## Overview

Redis-based scheduler and Message queue Spider for Scrapy, Provide more flexible and practical ways for Scrapy.

For more information [Document](https://ycvbcvfu.github.io/SoDo) [中文文档](https://ycvbcvfu.github.io/SoDo/#/zh-cn/):

## Requirements

  • Python 3

  • Scrapy >=1.6

## Features

  • Distributed

> Redis-based scheduler

  • Hooks feature

> new feature for Scrapy projects

  • Flexible parameter configuration

> Kafka , Redis client parameter set by yourself.

  • Filter interface

> Built-in BloomFiler and OriginalFilter

## Installation

`python pip install -U sodo -i https://pypi.org/simple `

## Contribution guidelines

If you want to contribute to SoDo, please see [CONTRIBUTING Doc](https://github.com/ycvbcvfu/SoDo/blob/master/CONTRIBUTING.md)

## For more information

### wechat

![wechat](docs/picture/wechat.JPG)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

SoDo-1.0.1.tar.gz (13.5 kB view details)

Uploaded Source

Built Distribution

SoDo-1.0.1-py2.py3-none-any.whl (19.5 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file SoDo-1.0.1.tar.gz.

File metadata

  • Download URL: SoDo-1.0.1.tar.gz
  • Upload date:
  • Size: 13.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.5

File hashes

Hashes for SoDo-1.0.1.tar.gz
Algorithm Hash digest
SHA256 aeab9c23f30bafee45c7a10c600cfbe31778fb9ec3a3746472853f49dab790fa
MD5 e0ef3201bd85cdac484fb331445f8aef
BLAKE2b-256 86da791b6f63b33f72e5a7f200f634242cb0f5623868d484b07ee6a9f56ffb26

See more details on using hashes here.

File details

Details for the file SoDo-1.0.1-py2.py3-none-any.whl.

File metadata

  • Download URL: SoDo-1.0.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 19.5 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.5

File hashes

Hashes for SoDo-1.0.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 720338affb229351aab65f71dde89055412b92740b3c76445da593e4cce568f7
MD5 e1751af9aac548ea845a0976a72bc07e
BLAKE2b-256 9e0be108fef7c2665b64bceaf3e4924c53cfaa4e600b3c24eafae3e634495072

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page