ScrapyCloud HubStorage frontier backend for Frontera
Project description
HCF (HubStorage Crawl Frontier) Backend for Frontera
When used with scrapy, use it with Scrapy Scheduler provided by scrapy-frontera. Scrapy scheduler provided by Frontera is not supported. scrapy-frontera is a scrapy scheduler which allows to use frontera backends, like the present one, with scrapy projects.
See specific usage instructions at module and class docstrings at backend.py. Some examples of usage can be seen in the scrapy-frontera README.
And a complete tutorial for using hcf-backend with ScrapyCloud workflows is available at shub-workflow Tutorial. shub-workflow is a framework for defining workflows of spiders and scripts running over ScrapyCloud. This is a strongly recommended lecture, because it documents the integration of different tools which together provide the best benefit.
Package also provides a convenient command line tool for hubstorage frontier handling and manipulation: hcfpal.py. It supports dumping, count, deletion, moving, listing, etc. See command line help for usage.
Another tool provided is crawlmanager.py. It facilitates the scheduling of consumer spider jobs. Examples of usage are also available in the already mentioned shub-workflow Tutorial.
Installation
pip install hcf-backend
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for hcf_backend-0.4.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47ade536f2a0bdadbc1b2afe6ab66118036f367d7db1bca48ab7523f7e71638f |
|
MD5 | 39c253a163f416fcdc99d58d931b5d97 |
|
BLAKE2b-256 | c2490fe898dc30d307c573621235fdf3f7685b40858f1b6d7757d5a8aca969ef |