HCF (HubStorage Crawl Frontier) Backend for Frontera
When used with scrapy, use it with Scrapy Scheduler provided by scrapy-frontera. Scrapy scheduler provided
by Frontera is not supported. scrapy-frontera is a scrapy scheduler which allows to use frontera backends,
like the present one, with scrapy projects.
See specific usage instructions at module and class docstrings at backend.py. Some examples of usage can be seen in the scrapy-frontera README.
A complete tutorial for using hcf-backend with ScrapyCloud workflows is available at
shub-workflow Tutorial: Managing Hubstorage Crawl Frontiers. shub-workflow is a framework for
defining workflows of spiders and scripts running over ScrapyCloud. This is a strongly recommended lecture, because it documents the integration of different tools
which together provide the best benefit.
Package also provides a convenient command line tool for hubstorage frontier handling and manipulation: hcfpal.py. It supports dumping, count, deletion, moving, listing, etc. See command line help for usage.
Another provided tool is crawlmanager.py. It facilitates the
scheduling of consumer spider jobs. Examples of usage are also available in the already mentioned shub-workflow Tutorial.
Installation
pip install hcf-backend
Development environment setup
For hcf-backend developers, Pipfile files are provided for a development environment.
Run:
$ pipenv install --dev
$ pipenv shell
$ cp .envtemplate .env
and edit .env accordingly
Release files for hcf-backend 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hcf_backend-0.6.1.tar.gz | 15.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hcf_backend-0.6.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 30.2 kB
Release files / hcf_backend-0.6.1.tar.gz
| Download URL | hcf_backend-0.6.1.tar.gz |
|---|---|
| Size | 15.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0a72b1ae30137ba9aa473936eabf0246efc8f0bf557fe37a6cc64af6c19ae776
|
|
BLAKE2b-256 checksum How to use checksums |
ea4fa4cf8d5afdca5d32d1324fd50eb831049c6bd71759eb0a17e564d4c6f2cb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / hcf_backend-0.6.1-py3-none-any.whl
| Download URL | hcf_backend-0.6.1-py3-none-any.whl |
|---|---|
| Size | 14.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
37a426238b36a7ca695432abfeabc45879630aaec5cd3d4dcb84492eab8754b8
|
|
BLAKE2b-256 checksum How to use checksums |
bdf16f957ce9423abbb60fd38281d1681e1c21d4bd7edc8f1656caab826ac449
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
