Behoof Scrapy Cluster Template

Overview

The bhfutils package is a collection of utilities that are used by any spider of Behoof project.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7 or 3.6

Installation

Inside a virtualenv, run pip install -U bhfutils. This will install the latest version of the Behoof Scrapy Cluster Spider utilities. After that you can use special settings.py compatibal with scrapy cluster (template placed in crawler/setting_template.py)

Documentation

Full documentation for the bhfutils package does not exist

custom_cookies.py

The custom_cookies module is custom Cookies Middleware to pass our required cookies along but not persist between calls

distributed_scheduler.py

The distributed_scheduler module is scrapy request scheduler that utilizes Redis Throttled Priority Queues to moderate different domain scrape requests within a distributed scrapy cluster

redis_domain_max_page_filter.py

The redis_domain_max_page_filter module is redis-based max page filter. This filter is applied per domain. Using this filter the maximum number of pages crawled for a particular domain is bounded

redis_dupefilter.py

The redis_dupefilter module is redis-based request duplication filter

redis_global_page_per_domain_filter.py

The redis_global_page_per_domain_filter module is redis-based request number filter When this filter is enabled, all crawl jobs have GLOBAL_PAGE_PER_DOMAIN_LIMIT as a hard limit of the max pages they are allowed to crawl for each individual spiderid+domain+crawlid combination.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.3.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vvteplygin from gravatar.com vvteplygin

Credits

Author: Teplygin Vladimir

License

MIT

Provides Extra

test all docs lint

Tags

behoof scrapy-cluster utilities

Report project as malware

Release files for bhfutils 0.2.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bhfutils 0.2.6
File Size Uploaded
bhfutils-0.2.6.tar.gz 92.3 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for bhfutils 0.2.6
File Interpreter ABI Platform
bhfutils-0.2.6-py3-none-any.whl 125.9 kB Python 3 none any Details

Total release size: 218.2 kB

Release files / bhfutils-0.2.6.tar.gz

Download URL bhfutils-0.2.6.tar.gz
Size 92.3 kB
Tags Source
SHA-256 checksum
How to use checksums 		0bef4f48bcb67b3cf4ce61bea4a65cbe54fdd4a4b575efa0e524dc42f532771d
BLAKE2b-256 checksum
How to use checksums 		507284718dbef9190054537b9744295916920256955bf95905e471029f4f15ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / bhfutils-0.2.6-py3-none-any.whl

Download URL bhfutils-0.2.6-py3-none-any.whl
Size 125.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		5a054aa97960a2fbf14301adeb7a42da47086b9f0913ae41a58a690cfca08039
BLAKE2b-256 checksum
How to use checksums 		b1820315edf1721aefd1676aacb4163d127e94da87149e19a2562fa17ca24ed1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/6.2.0 CPython/3.9.6

Release history Release notifications | RSS feed

0.3.0

2 release files

0.2.99

2 release files

0.2.8

2 release files

0.2.7

2 release files

This release

0.2.6 This release

2 release files

0.2.4

2 release files

0.2.3

2 release files

0.2.2

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.22

2 release files

0.1.21

2 release files

0.1.20

2 release files

0.1.19

2 release files

0.1.18

2 release files

0.1.17

2 release files

0.1.16

2 release files

0.1.15

2 release files

0.1.14

2 release files

0.1.13

2 release files

0.1.12

2 release files

0.1.11

2 release files

0.1.10

2 release files

0.1.9

2 release files

0.1.8

2 release files

0.1.7

2 release files

0.1.5

1 release file

0.1.4

1 release file

0.1.3

1 release file

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.101

2 release files

0.0.100

2 release files

0.0.99

2 release files

0.0.98

2 release files

0.0.97

2 release files

0.0.96

2 release files

0.0.95

2 release files

0.0.94

2 release files

0.0.93

2 release files

0.0.92

2 release files

0.0.91

2 release files

0.0.90

2 release files

0.0.89

2 release files

0.0.88

2 release files

0.0.87

2 release files

0.0.86

2 release files

0.0.85

2 release files

0.0.84

2 release files

0.0.83

2 release files

0.0.82

2 release files

0.0.81

1 release file

0.0.80

1 release file

0.0.79

1 release file

0.0.78

1 release file

0.0.77

1 release file

0.0.76

1 release file

0.0.75

1 release file

0.0.74

1 release file

0.0.73

1 release file

0.0.72

1 release file

0.0.71

1 release file

0.0.70

1 release file

0.0.69

1 release file

0.0.68

1 release file

0.0.67

1 release file

0.0.66

1 release file

0.0.65

1 release file

0.0.64

1 release file

0.0.63

1 release file

0.0.62

1 release file

0.0.61

1 release file

0.0.60

1 release file

0.0.59

1 release file

0.0.58

1 release file

0.0.57

1 release file

0.0.56

1 release file

0.0.55

1 release file

0.0.54

1 release file

0.0.53

1 release file

0.0.52

1 release file

0.0.51

1 release file

0.0.50

1 release file

0.0.49

1 release file

0.0.48

1 release file

0.0.47

1 release file

0.0.46

1 release file

0.0.45

1 release file

0.0.44

1 release file

0.0.43

1 release file

0.0.42

1 release file

0.0.41

1 release file

0.0.40

1 release file

0.0.39

1 release file

0.0.38

1 release file

0.0.37

1 release file

0.0.36

1 release file

0.0.35

1 release file

0.0.34

1 release file

0.0.33

1 release file

0.0.32

1 release file

0.0.31

1 release file

0.0.30

1 release file

0.0.15

1 release file

0.0.14

1 release file

0.0.13

1 release file

0.0.12

1 release file

0.0.11

1 release file

0.0.10

1 release file

0.0.9

1 release file

0.0.8

1 release file

0.0.7

1 release file

0.0.6

1 release file

0.0.5

1 release file

0.0.4

1 release file

0.0.2

1 release file

0.0.1

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page