crawler-utils

Scrapy utils for Modis crawlers projects.

MongoDB

Some utils connected with mongodb.

MongoDBPipeline - pipeline for saving items in mongodb.

Params:

  • MONGODB_SERVER - address of mongodb database.
  • MONGODB_PORT - port of mongodb database.
  • MONGODB_DB - database where to save data.
  • MONGODB_USERNAME - username for authentication in MONGODB_DB database.
  • MONGODB_PWD - password for authentication.
  • DEFAULT_MONGODB_COLLECTION - default collection where to save data (default value is test).
  • MONGODB_COLLECTION_KEY - key of item which identifies items collection name (MONGO_COLLECTION) where to save item (default value is collection).
  • MONGODB_UNIQUE_KEY - key of item which identifies item

Kafka

Some utils connected with kafka.

KafkaPipeline - pipeline for pushing items into kafka.

Pipeline outputs data into stream with name {RESOURCE_TAG}.{DATA_TYPE}. Where RESOURCE_TAG is tag of resource from which data is crawled and DATA_TYPE is type of data crawled: data, post, comment, like, user, friend, share, member, news, community.

Params:

  • KAFKA_ADDRESS - address of kafka broker.
  • KAFKA_KEY - key of item which is put into kafka record key.
  • KAFKA_RESOURCE_TAG_KEY - key of item which identifies item RESOURCE_TAG (default value is platform)
  • KAFKA_DEFAULT_RESOURCE_TAG - default RESOURCE_TAG for crawled items without KAFKA_RESOURCE_TAG_KEY (default value is crawler)
  • KAFKA_DATA_TYPE_KEY - key of item from which identifies item DATA_TYPE (default value is type).
  • KAFKA_DEFAULT_DATA_TYPE - default DATA_TYPE for crawled items without KAFKA_DATA_TYPE_KEY (default value is data).
  • KAFKA_COMPRESSION_TYPE - type of data compression in kafka for example gzip.

OpenSearch

OpenSearchRequestsDownloaderMiddleware transforms request-response pair into an item, and then sends it to the OpenSearch.

Settings:

`OPENSEARCH_REQUESTS_SETTINGS` - dict specifying OpenSearch client connections:
    "hosts": Optional[str | list[str]] = "localhost:9200" - hosts with opensearch endpoint,
    "timeout": Optional[int] = 60 - timeout of connections,
    "http_auth": Optional[tuple[str, str]] = None - HTTP authentication if needed,
    "port": Optional[int] = 443 - access port if not specified in hosts,
    "use_ssl": Optional[bool] = True - usage of SSL,
    "verify_certs": Optional[bool] = False - verifying certificates,
    "ssl_show_warn": Optional[bool] = False - show SSL warnings,
    "ca_certs": Optional[str] = None - CA certificate path,
    "client_key": Optional[str] = None - client key path,
    "client_cert": Optional[str] = None - client certificate path,
    "buffer_length": Optional[int] = 500 - number of items in OpenSearchStorage's buffer.

`OPENSEARCH_REQUESTS_INDEX`: Optional[str] = "scrapy-job-requests" - index in OpenSearch.

See an example in examples/opensearch.

CaptchaDetection

Captcha detection middleware for scrapy crawlers. It gets the HTML code from the response (if present), sends it to the captcha detection web-server and logs the result.

If you don't want to check exact response if it has captcha, provide meta-key dont_check_captcha with True value.

The middleware must be set up with higher precedence (lower number) than RetryMiddleware:

DOWNLOADER_MIDDLEWARES = {
    "crawler_utils.CaptchaDetectionDownloaderMiddleware": 549,  # By default, RetryMiddleware has 550
}

Middleware settings:

  • ENABLE_CAPTCHA_DETECTOR: bool = True. Whether to enable captcha detection.
  • CAPTCHA_SERVICE_URL: str. For an example: http://127.0.0.1:8000

Sentry logging

You may want to log exceptions during crawling to your Sentry. Use the crawler_utils.sentry_logging.SentryLoggingExtension for this. Note that sentry_sdk wants to be loaded as earlier as possible. To satisfy this condition make the extension with negative order:

EXTENSIONS = {
    # Load SentryLogging extension before other extensions.
    "crawler_utils.sentry_logging.SentryLoggingExtension": -1,
}

Settings:

SENTRY_DSN: str - Sentry's DSN, where to send events.
SENTRY_SAMPLE_RATE: float = 1.0 - sample rate for error events. Must be in range from 0.0 to 1.0.
SENTRY_TRACES_SAMPLE_RATE: float = 1.0 - the percentage chance a given transaction will be sent to Sentry.
SENTRY_ATTACH_STACKTRACE: bool = False - whether to attach stacktrace for error events.
SENTRY_MAX_BREADCRUMBS: int = 10 - max breadcrumbs to capture with Sentry.

For an example, check examples/sentry_logging.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ispras_crawlers from gravatar.com ispras_crawlers

Credits

Author: Varlamov

License

BSD License, Other/Proprietary License (BSD)

Requires

Python <4.0, >=3.9

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

modis_crawler_utils-0.3.58.tar.gz (74.8 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

modis_crawler_utils-0.3.58-py3-none-any.whl (108.6 kB view details)

Uploaded Python 3

File details

Details for the file modis_crawler_utils-0.3.58.tar.gz.

File metadata

  • Download URL: modis_crawler_utils-0.3.58.tar.gz
  • Upload date:
  • Size: 74.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.23

File hashes

Hashes for modis_crawler_utils-0.3.58.tar.gz
Algorithm Hash digest
SHA256 4d41b4b4a1c7accec11339e0f84c06e4014544d485ef5a0734e647de4ae2931c
MD5 ea1183119a0910a67ca3dddfa7e80e7e
BLAKE2b-256 54568b4757f4e1cf532a8965e94e312998da257f1652138efec1e02803823925

See more details on using hashes here.

File details

Details for the file modis_crawler_utils-0.3.58-py3-none-any.whl.

File metadata

File hashes

Hashes for modis_crawler_utils-0.3.58-py3-none-any.whl
Algorithm Hash digest
SHA256 69909cf95e0c78be655c1117f1cf134490887cea582cca44c0175aa8a4d9ffdd
MD5 5b9d06c8cee3d7789367f85cb26692d2
BLAKE2b-256 c6e37b0f7d43e2819c5de02f8e28d1d3aa218adf0cd77de79d562e0e1494ee3f

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

0.3.58 This release

2 files

0.3.57

2 files

0.3.56

2 files

0.3.55

2 files

0.3.54

2 files

0.3.53

2 files

0.3.52

2 files

0.3.51

2 files

0.3.50

2 files

0.3.49

2 files

0.3.48

2 files

0.3.47

2 files

0.3.46

2 files

0.3.45

2 files

0.3.44

2 files

0.3.43

2 files

0.3.42

2 files

0.3.41

2 files

0.3.40

2 files

0.3.39

2 files

0.3.38

2 files

0.3.37

2 files

0.3.36

2 files

0.3.35

2 files

0.3.34

2 files

0.3.33

2 files

0.3.32

2 files

0.3.31

2 files

0.3.30

2 files

0.3.29

2 files

0.3.28

2 files

0.3.27

2 files

0.3.25

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

0.2.84

2 files

0.2.83

2 files

0.2.82

2 files

0.2.81

2 files

0.2.80

2 files

0.2.79

2 files

0.2.78

2 files

0.2.77

2 files

0.2.76

2 files

0.2.74

2 files

0.2.73

2 files

0.2.71

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page