Skip to main content

Heavy-job queue processor

Project description

Heavy-job queue processor


pip install shire

Create config file

Run shire-cli create_config, follow instructions.

An example of the shire.cfg:

redis_url = redis://
db_url = postgres://peewee:123@

check_time = 5

venv_path = /path/to/your/virtualenv
sys_path = /path/to/your/project:/path/to/external/library
host = default

per_pool = 1
log_max_size = 104857600
log_directory = logs
max_logs = 5
default_log = default.log

check_time = 30

limits_update_time = 60
check_time = 1

Queue management

Interactive shire config creation
shire-cli -c /path/to/your/shire.cfg create_config
Run whip (Enqueue jobs to pools)
shire-cli -c /path/to/your/shire.cfg run_whip
Run scribe (Shire job logs writer)
shire-cli -c /path/to/your/shire.cfg run_scribe
Run hostler (Shire failed jobs restarter)
shire-cli -c /path/to/your/shire.cfg run_hostler
Run pool (Shire job executor)
shire-cli -c /path/to/your/shire.cfg run_pool –name=pool_name
Run multiple pools with one master-process
shire-cli -c /path/to/your/shire.cfg start_pools –names=pool_name,another_pool


Run shire
shire-cli -c /path/to/your/shire.cfg run_whip shire-cli -c /path/to/your/shire.cfg run_pool –name=default

Create example jobs module:

from shire.job import Job
import requests

def save(text_len):
    # save - your external function to save result data. Shire not save any results itself.

class CountWordsAtUrl(Job):
    def run(url=None):
        resp = requests.get(url)

Create example configuration module:

from shire.config import Config as ShireConfig
conf = ShireConfig()

Enqueue job:

from my_jobs import CountWordsAtUrl
from shire_conf import conf
CountWordsAtUrl.delay(conf, pool='default', kwargs={'url': ''})

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shire-0.2.1.tar.gz (20.8 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page