Behoof Scrapy Cluster Template

Overview

The bhfutils package is a collection of utilities that are used by any spider of Behoof project.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7 or 3.6

Installation

Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation

Full documentation for the bhfutils package does not exist

custom_cookies.py

The custom_cookies module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py

The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py

The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded

redis_dupefilter.py

The redis_dupefilter module is redis-based request duplication filter

redis_global_page_per_domain_filter.py

The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Release files for bhfutils 0.2.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bhfutils 0.2.8
File Size Uploaded
bhfutils-0.2.8.tar.gz 93.4 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for bhfutils 0.2.8
File Interpreter ABI Platform
bhfutils-0.2.8-py3-none-any.whl 127.0 kB Python 3 none any Details

Total release size: 220.4 kB

Release files / bhfutils-0.2.8.tar.gz

Download URL bhfutils-0.2.8.tar.gz
Size 93.4 kB
Tags Source
SHA-256 checksum
How to use checksums 		7ed663f91aa26744476256d3a3b63a0819b7ec385ae3da38117eaf223104c360
BLAKE2b-256 checksum
How to use checksums 		c90c00627f906aeafd5969dbfc3bbb3af6d737f0c3150373d43e5ec381a122de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / bhfutils-0.2.8-py3-none-any.whl

Download URL bhfutils-0.2.8-py3-none-any.whl
Size 127.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		91e8926a0d134d74ec7e1fc5ec8184d487c22b898a744f23dfdead2a0fe63acb
BLAKE2b-256 checksum
How to use checksums 		f8ecaea44e7ecb909333dd8a7d2854d26037eaec26737946593293354e613d31
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release history Release notifications | RSS feed

0.3.0

2 release files

0.2.99

2 release files

This release

0.2.8 This release

2 release files

0.2.7

2 release files

0.2.6

2 release files

0.2.4

2 release files

0.2.3

2 release files

0.2.2

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.22

2 release files

0.1.21

2 release files

0.1.20

2 release files

0.1.19

2 release files

0.1.18

2 release files

0.1.17

2 release files

0.1.16

2 release files

0.1.15

2 release files

0.1.14

2 release files

0.1.13

2 release files

0.1.12

2 release files

0.1.11

2 release files

0.1.10

2 release files

0.1.9

2 release files

0.1.8

2 release files

0.1.7

2 release files

0.1.5

1 release file

0.1.4

1 release file

0.1.3

1 release file

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.101

2 release files

0.0.100

2 release files

0.0.99

2 release files

0.0.98

2 release files

0.0.97

2 release files

0.0.96

2 release files

0.0.95

2 release files

0.0.94

2 release files

0.0.93

2 release files

0.0.92

2 release files

0.0.91

2 release files

0.0.90

2 release files

0.0.89

2 release files

0.0.88

2 release files

0.0.87

2 release files

0.0.86

2 release files

0.0.85

2 release files

0.0.84

2 release files

0.0.83

2 release files

0.0.82

2 release files

0.0.81

1 release file

0.0.80

1 release file

0.0.79

1 release file

0.0.78

1 release file

0.0.77

1 release file

0.0.76

1 release file

0.0.75

1 release file

0.0.74

1 release file

0.0.73

1 release file

0.0.72

1 release file

0.0.71

1 release file

0.0.70

1 release file

0.0.69

1 release file

0.0.68

1 release file

0.0.67

1 release file

0.0.66

1 release file

0.0.65

1 release file

0.0.64

1 release file

0.0.63

1 release file

0.0.62

1 release file

0.0.61

1 release file

0.0.60

1 release file

0.0.59

1 release file

0.0.58

1 release file

0.0.57

1 release file

0.0.56

1 release file

0.0.55

1 release file

0.0.54

1 release file

0.0.53

1 release file

0.0.52

1 release file

0.0.51

1 release file

0.0.50

1 release file

0.0.49

1 release file

0.0.48

1 release file

0.0.47

1 release file

0.0.46

1 release file

0.0.45

1 release file

0.0.44

1 release file

0.0.43

1 release file

0.0.42

1 release file

0.0.41

1 release file

0.0.40

1 release file

0.0.39

1 release file

0.0.38

1 release file

0.0.37

1 release file

0.0.36

1 release file

0.0.35

1 release file

0.0.34

1 release file

0.0.33

1 release file

0.0.32

1 release file

0.0.31

1 release file

0.0.30

1 release file

0.0.15

1 release file

0.0.14

1 release file

0.0.13

1 release file

0.0.12

1 release file

0.0.11

1 release file

0.0.10

1 release file

0.0.9

1 release file

0.0.8

1 release file

0.0.7

1 release file

0.0.6

1 release file

0.0.5

1 release file

0.0.4

1 release file

0.0.2

1 release file

0.0.1

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page