scrapy_redis use bloomfilter
Project description
Scrapy-Redis-BloomFilter
这是个scrapy_redis的布隆过滤器版本,与https://github.com/Python3WebSpider/ScrapyRedisBloomFilter
不同的是,该项目使用redis里的布隆过滤器,而不是使用bit来实现
必要条件
需要redis加载了布隆过滤器的插件,默认安装的redis是没有加载的 具体请看:https://redis.io/docs/stack/bloom/quick_start/
安装
使用pip: pip install scrapy-redis-bf
使用
在scrapy项目里的 settings.py
添加如下设置:
SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"
# 格式:redis://[:password@]host[:port][/database][?[timeout=timeout[d|h|m|s|ms|us|ns]][&database=database]]
REDIS_URL = 'redis://localhost:6379'
# 错误率
BLOOMFILTER_ERRORRATE = 0.001
# 去重量
BLOOMFILTER_CAPACITY = 10000
测试
下载该项目,然后运行里面的test spider即可
Github
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy_redis_bf-0.0.9.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for scrapy_redis_bf-0.0.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6302b176ef0afd808e3839e0c53202471be8c464f7aa2efe07e1738f9e191839 |
|
MD5 | 6f1bac4709f50f2c383928c4213576c5 |
|
BLAKE2b-256 | 2a737b45f1b8fd3425517b908df4908947048c65f9148a0ecab691339b34eb46 |