A distributed crawler component based on scrapy_redis which can specify the expiration time of fingerprints in dupefilter.
Project description
The author of this package has not provided a project description
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for scrapy-redis-expiredupefilter-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 290af25d5e8d6e0bde73df553dedf923fc5b280ba5ab4540293be8ca9f329b2f |
|
MD5 | 154a2f0a18b8bf9316789d32370e0a22 |
|
BLAKE2b-256 | c6aa4b4b42641d4a266d3b442651d69f793bcb2691eef4d320464d8274b60df6 |