A distributed crawler component based on scrapy_redis which can specify the expiration time of fingerprints in dupefilter.
Project description
The author of this package has not provided a project description
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for scrapy-redis-expiredupefilter-0.1.99.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54d2a2e6c6b6e0d541bb714ed5a49c930f9660230955224c0df1db1a4fb6d004 |
|
MD5 | b01079502de224c598b0b774e67f5348 |
|
BLAKE2b-256 | d08438eddd31d96754a158b587dc7e70a8b96cbee57c4c7394ceb7202c9634da |