Behoof Scrapy Cluster Template

Overview

The bhfutils package is a collection of utilities that are used by any spider of Behoof project.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7 or 3.6

Installation

Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation

Full documentation for the bhfutils package does not exist

custom_cookies.py

The custom_cookies module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py

The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py

The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded

redis_dupefilter.py

The redis_dupefilter module is redis-based request duplication filter

redis_global_page_per_domain_filter.py

The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Release files for bhfutils 0.2.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bhfutils 0.2.7
File Size Uploaded
bhfutils-0.2.7.tar.gz 92.8 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for bhfutils 0.2.7
File Interpreter ABI Platform
bhfutils-0.2.7-py3-none-any.whl 126.4 kB Python 3 none any Details

Total release size: 219.2 kB

Release files / bhfutils-0.2.7.tar.gz

Download URL bhfutils-0.2.7.tar.gz
Size 92.8 kB
Tags Source
SHA-256 checksum
How to use checksums 		cf0bb80788277346ba7ee8bec8f1c2852b29191ca87659b44e22c6a78e500a74
BLAKE2b-256 checksum
How to use checksums 		28d47c974c1bccd01cae8374db53429d25c5b9a0e8c1f56826bdc3ddcc461014
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / bhfutils-0.2.7-py3-none-any.whl

Download URL bhfutils-0.2.7-py3-none-any.whl
Size 126.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		f332a361737ac0626b93f502971e8e00b79c111a53f1cdd0f6975084c530538c
BLAKE2b-256 checksum
How to use checksums 		06ccf3d65fe94060748b9529e5c2fc90771866a4abc62cb4f9e117701b94dd8a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release history Release notifications | RSS feed

0.3.0

2 release files

0.2.99

2 release files

0.2.8

2 release files

This release

0.2.7 This release

2 release files

0.2.6

2 release files

0.2.4

2 release files

0.2.3

2 release files

0.2.2

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.22

2 release files

0.1.21

2 release files

0.1.20

2 release files

0.1.19

2 release files

0.1.18

2 release files

0.1.17

2 release files

0.1.16

2 release files

0.1.15

2 release files

0.1.14

2 release files

0.1.13

2 release files

0.1.12

2 release files

0.1.11

2 release files

0.1.10

2 release files

0.1.9

2 release files

0.1.8

2 release files

0.1.7

2 release files

0.1.5

1 release file

0.1.4

1 release file

0.1.3

1 release file

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.101

2 release files

0.0.100

2 release files

0.0.99

2 release files

0.0.98

2 release files

0.0.97

2 release files

0.0.96

2 release files

0.0.95

2 release files

0.0.94

2 release files

0.0.93

2 release files

0.0.92

2 release files

0.0.91

2 release files

0.0.90

2 release files

0.0.89

2 release files

0.0.88

2 release files

0.0.87

2 release files

0.0.86

2 release files

0.0.85

2 release files

0.0.84

2 release files

0.0.83

2 release files

0.0.82

2 release files

0.0.81

1 release file

0.0.80

1 release file

0.0.79

1 release file

0.0.78

1 release file

0.0.77

1 release file

0.0.76

1 release file

0.0.75

1 release file

0.0.74

1 release file

0.0.73

1 release file

0.0.72

1 release file

0.0.71

1 release file

0.0.70

1 release file

0.0.69

1 release file

0.0.68

1 release file

0.0.67

1 release file

0.0.66

1 release file

0.0.65

1 release file

0.0.64

1 release file

0.0.63

1 release file

0.0.62

1 release file

0.0.61

1 release file

0.0.60

1 release file

0.0.59

1 release file

0.0.58

1 release file

0.0.57

1 release file

0.0.56

1 release file

0.0.55

1 release file

0.0.54

1 release file

0.0.53

1 release file

0.0.52

1 release file

0.0.51

1 release file

0.0.50

1 release file

0.0.49

1 release file

0.0.48

1 release file

0.0.47

1 release file

0.0.46

1 release file

0.0.45

1 release file

0.0.44

1 release file

0.0.43

1 release file

0.0.42

1 release file

0.0.41

1 release file

0.0.40

1 release file

0.0.39

1 release file

0.0.38

1 release file

0.0.37

1 release file

0.0.36

1 release file

0.0.35

1 release file

0.0.34

1 release file

0.0.33

1 release file

0.0.32

1 release file

0.0.31

1 release file

0.0.30

1 release file

0.0.15

1 release file

0.0.14

1 release file

0.0.13

1 release file

0.0.12

1 release file

0.0.11

1 release file

0.0.10

1 release file

0.0.9

1 release file

0.0.8

1 release file

0.0.7

1 release file

0.0.6

1 release file

0.0.5

1 release file

0.0.4

1 release file

0.0.2

1 release file

0.0.1

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page