ScrapyCloud HubStorage frontier backend for Frontera
Project description
HCF (HubStorage Frontier) Backend for Frontera
When used with scrapy, use it with Scrapy Scheduler provided by scrapy-frontera. Scrapy scheduler provided by Frontera is not supported
See usage instructions at module and class docstrings at backend.py.
Package also provides a convenient command line tool for hubstorage frontier handling and manipulation: hcfpal.py. It supports dumping, count, deletion, moving, listing, etc.
Another tool provided is crawlmanager.py. It facilitates the scheduling of consumer spider jobs.
Installation
pip install hcf-backend
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hcf-backend-0.3.3.tar.gz
(10.2 kB
view hashes)
Built Distribution
Close
Hashes for hcf_backend-0.3.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ab3115fa4c202dfa04bb34bb54185a14013ad347886bf3fe0a316b4b24b6699 |
|
MD5 | 9ef7ef2ef6528d9592d4021a0b03dd0c |
|
BLAKE2b-256 | 426be7f0fe2e99d9488233fe7ee0b2a050ef0ed4dab7cf3ca605b45e19c3e8b7 |