Skip to main content

Utilities that are used by any spider of Behoof project

Project description

Overview

The bhfutils package is a collection of utilities that are used by any spider of Behoof project.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7 or 3.6

Installation

Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation

Full documentation for the bhfutils package does not exist

custom_cookies.py

The custom_cookies module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py

The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py

The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded

redis_dupefilter.py

The redis_dupefilter module is redis-based request duplication filter

redis_global_page_per_domain_filter.py

The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bhfutils-0.0.92.tar.gz (85.4 kB view details)

Uploaded Source

Built Distribution

bhfutils-0.0.92-py3-none-any.whl (118.4 kB view details)

Uploaded Python 3

File details

Details for the file bhfutils-0.0.92.tar.gz.

File metadata

  • Download URL: bhfutils-0.0.92.tar.gz
  • Upload date:
  • Size: 85.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.0

File hashes

Hashes for bhfutils-0.0.92.tar.gz
Algorithm Hash digest
SHA256 2146013c664652f86f1fdaf893f015fdffe8188ded2fc0f7103371d7ce70b9f8
MD5 4614076b4355ad8241babef32252b08f
BLAKE2b-256 bf2e7b780e32247630d850bc0d002bc3787e04c055d5b543b2e3100853d8dfa9

See more details on using hashes here.

File details

Details for the file bhfutils-0.0.92-py3-none-any.whl.

File metadata

  • Download URL: bhfutils-0.0.92-py3-none-any.whl
  • Upload date:
  • Size: 118.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.0

File hashes

Hashes for bhfutils-0.0.92-py3-none-any.whl
Algorithm Hash digest
SHA256 7c21095328a3618ed085c9fb2cd01496049f1428b54315386b1c6cd2d67c16ce
MD5 1432bac9c2812a9a40f2c67856a4749f
BLAKE2b-256 666f61fd808943bb972ed5302af556686663554903bac2ecf6c98e7dff1e9f9a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page