Redis Cluster for Scrapy.
Project description
本项目基于原项目 scrpy-redis 进行修改,修改内容如下:
添加了 Redis 哨兵连接支持
添加了 Redis 集群连接支持
TODO 去重
配置示例
pip install scrapy-redis-sentinel --user
原版本的所有配置都支持, 优先级:哨兵模式 > 集群模式 > 单机模式
# ----------------------------------------Redis 单机模式-------------------------------------
# Redis 单机地址
REDIS_HOST = "172.25.2.25"
REDIS_PORT = 6379
# REDIS 单机模式配置参数
REDIS_PARAMS = {
"password": "xxxx",
"db": 0
}
# ----------------------------------------Redis 哨兵模式-------------------------------------
# Redis 哨兵地址
REDIS_SENTINELS = [
('172.25.2.25', 26379),
('172.25.2.26', 26379),
('172.25.2.27', 26379)
]
# REDIS_SENTINEL_PARAMS 哨兵模式配置参数。
REDIS_SENTINEL_PARAMS= {
"service_name":"mymaster",
"password": "xxxx",
"db": 0
}
# ----------------------------------------Redis 集群模式-------------------------------------
# Redis 集群地址
REDIS_MASTER_NODES = [
{"host": "172.25.2.25", "port": "6379"},
{"host": "172.25.2.26", "port": "6379"},
{"host": "172.25.2.27", "port": "6379"},
]
# REDIS_CLUSTER_PARAMS 集群模式配置参数
REDIS_CLUSTER_PARAMS= {
"password": "xxxx",
"db": 0
}
# ----------------------------------------Scrapy 其他参数-------------------------------------
# 在 redis 中保持 scrapy-redis 用到的各个队列,从而允许暂停和暂停后恢复,也就是不清理 redis queues
SCHEDULER_PERSIST = True
# 调度队列
SCHEDULER = "scrapy_redis_sentinel.scheduler.Scheduler"
# 去重
DUPEFILTER_CLASS = "scrapy_redis_sentinel.dupefilter.RFPDupeFilter"
注:当使用集群时单机不生效
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for scrapy-redis-sentinel-0.1.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7cf05d6b0170f90892a375d5562c52c35a579e085a8af88891e153a2173092bd |
|
MD5 | 6f9d2893f7987f319f0ac7c36ceafe12 |
|
BLAKE2b-256 | 7c19cafa1c774de34088e86f017f687d84401d08f007808097026a42db831d9f |