Overview
The bhfutils package is a collection of utilities that are used by any spider of Behoof project.
Requirements
Unix based machine (Linux or OS X)
Python 2.7 or 3.6
Installation
Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)
Documentation
Full documentation for the bhfutils package does not exist
distributed_scheduler.py
The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster
redis_domain_max_page_filter.py
The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded
redis_dupefilter.py
The redis_dupefilter module is redis-based request duplication filter
redis_global_page_per_domain_filter.py
The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.
1 maintainer
Metadata
1 maintainer
Release files for bhfutils 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bhfutils-0.3.0.tar.gz | 91.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bhfutils-0.3.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 217.0 kB
Release files / bhfutils-0.3.0.tar.gz
| Download URL | bhfutils-0.3.0.tar.gz |
|---|---|
| Size | 91.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7587bab2822253f5584b63a4d80dca0fa6317778a3fc99d7f196ba360d567526
|
|
BLAKE2b-256 checksum How to use checksums |
3703e7623598b915f7bd2e753882db29a91f7997645a98b29b4d0a0174bba3f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Release files / bhfutils-0.3.0-py3-none-any.whl
| Download URL | bhfutils-0.3.0-py3-none-any.whl |
|---|---|
| Size | 125.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2984867bbfb5aa59aff5a7b19e440ba14ca4a90e17923aee39a383d2bf339472
|
|
BLAKE2b-256 checksum How to use checksums |
678741e7f7388c09370ef683357038f4f15126aaf969af6e563b023256cfef35
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
