Skip to main content

Utilities for Scrapy Cluster

Project description

Overview

The scutils package is a collection of utilities that are used by the Scrapy Cluster project. These utilities are agnostic enough that they can be used by any application.

Requirements

  • Unix based machine (Linux or OS X)

  • Python 2.7.x

Installation

Inside a virtualenv, run pip install -U scutils. This will install the latest version of the Scrapy Cluster Utilities. If you are running a Scrapy Cluster, scutils is included inside of the requirements.txt so there is no need to install it separately.

Documentation

Full documentation for the scutils package will be included as part of the Scrapy Cluster 1.1 release. Basic descriptions are below.

argparse_helper.py

The argparse_helper module is used to help print top level --help arguments from argparse when used with subparsers. Useful for running applications that have multiple combinations of subcommands and command line arguments.

log_factory.py

The log_factory module provides a standardized way for creating logs for multithreaded and concurrent process log data. It supports all log levels, stdout or to a file, and various output formats including JSON.

method_timer.py

The method_timer module provides a simple decorator that can be added to functions or methods requiring an execution timeout period.

redis_queue.py

The redis_queue module provides 3 core queue classes which use Redis as the place to store data. Includes FIFO, Stack, and Priority Queues.

redis_throttled_queue.py

The redis_throttled_queue module provides a throttled or moderated Redis queue structure that can be used to mitigate the number of pops from the queue within a given time frame.

settings_wrapper.py

The settings_wrapper module is a class the handles loading of default python application settings, which can then be overridden or added to by a local settings file. In the end provides a single dictionary object of all your loaded application settings.

stats_collector.py

The stats_collector module generates Redis based statistics based on time windows or in total. Statistics collection includes time windows, rolling time windows, counters, unique counters, hyperloglog counters, and bitmap counters.

zookeeper_watcher.py

The zookeeper_watcher module provides an easy way to tell an application that it’s watched Zookeeper file has changed. It also handles Zookeeper session disconnects and reconnects behind the scenes of your application.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scutils-0.0.3.tar.gz (16.2 kB view details)

Uploaded Source

Built Distribution

scutils-0.0.3-py2.7.egg (38.8 kB view details)

Uploaded Egg

File details

Details for the file scutils-0.0.3.tar.gz.

File metadata

  • Download URL: scutils-0.0.3.tar.gz
  • Upload date:
  • Size: 16.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for scutils-0.0.3.tar.gz
Algorithm Hash digest
SHA256 4652a91c865f919f1941cf20ba2357a31aef3c7ac13b3d731c6fe9810d9a0792
MD5 c394072fa2cb2c5e641a3c6bed7cd114
BLAKE2b-256 2a2cedd2e3b3fa62235cba76d0319fdd7463eddc5755a545e99c3e9684b03dd0

See more details on using hashes here.

File details

Details for the file scutils-0.0.3-py2.7.egg.

File metadata

  • Download URL: scutils-0.0.3-py2.7.egg
  • Upload date:
  • Size: 38.8 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for scutils-0.0.3-py2.7.egg
Algorithm Hash digest
SHA256 8ffeda229e1e61df200b60bf6ce9069eda2142743bf564a516faaf2cbace4e98
MD5 c96cce08db52d0c0aebe166e3f4f3e0e
BLAKE2b-256 d7db5593518fe77e35107810617ad5a45827f8988fc619aababdc0ebf3abbe01

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page