Skip to main content

Utilities that are used by any spider of Behoof project

Project description

Overview

The bhfutils package is a collection of utilities that are used by any spider of Behoof project.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7 or 3.6

Installation

Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation

Full documentation for the bhfutils package does not exist

custom_cookies.py

The custom_cookies module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py

The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py

The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded

redis_dupefilter.py

The redis_dupefilter module is redis-based request duplication filter

redis_global_page_per_domain_filter.py

The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bhfutils-0.1.10.tar.gz (86.3 kB view details)

Uploaded Source

Built Distribution

bhfutils-0.1.10-py3-none-any.whl (119.6 kB view details)

Uploaded Python 3

File details

Details for the file bhfutils-0.1.10.tar.gz.

File metadata

  • Download URL: bhfutils-0.1.10.tar.gz
  • Upload date:
  • Size: 86.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.6

File hashes

Hashes for bhfutils-0.1.10.tar.gz
Algorithm Hash digest
SHA256 051e7492491222d0f554d55c68f511353f826bd275709324fc92b35c433ecba6
MD5 257a1bddb633e0f2ed7c606288e72ff4
BLAKE2b-256 4541f96b3a90c5ee663d1627e5ee702064a59bc90e52a905bdad291a7cb90f12

See more details on using hashes here.

File details

Details for the file bhfutils-0.1.10-py3-none-any.whl.

File metadata

  • Download URL: bhfutils-0.1.10-py3-none-any.whl
  • Upload date:
  • Size: 119.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.6

File hashes

Hashes for bhfutils-0.1.10-py3-none-any.whl
Algorithm Hash digest
SHA256 4fbffde562d7ac53898f82e7c30a8d38e1d39c29ad42ef686b76f629e5d93469
MD5 f519406f96b3f3348eb429ab344c9e04
BLAKE2b-256 8b19b99591830ddaf5ae6112b87053621c3954b65d596c0fe68113f5040805bf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page