Skip to main content

Heavy-job queue processor

Project description

Heavy-job queue processor


pip install shire

Create config file

Run shire-cli create_config, follow instructions.

An example of the shire.cfg:

redis_url = redis://
db_url = postgres://peewee:123@

check_time = 5

venv_path = /path/to/your/virtualenv
sys_path = /path/to/your/project:/path/to/external/library
host = default

per_pool = 1
log_max_size = 104857600
log_directory = logs
max_logs = 5
default_log = default.log

check_time = 30

limits_update_time = 60
check_time = 1

Queue management

Interactive shire config creation

shire-cli -c /path/to/your/shire.cfg create_config

Run whip (Enqueue jobs to pools)

shire-cli -c /path/to/your/shire.cfg run_whip

Run scribe (Shire job logs writer)

shire-cli -c /path/to/your/shire.cfg run_scribe

Run hostler (Shire failed jobs restarter)

shire-cli -c /path/to/your/shire.cfg run_hostler

Run pool (Shire job executor)

shire-cli -c /path/to/your/shire.cfg run_pool –name=pool_name

Run multiple pools with one master-process

shire-cli -c /path/to/your/shire.cfg start_pools –names=pool_name,another_pool


Run shire

shire-cli -c /path/to/your/shire.cfg run_whip shire-cli -c /path/to/your/shire.cfg run_pool –name=default

Create example jobs module:

from shire.job import Job
import requests

def save(text_len):
    # save - your external function to save result data. Shire not save any results itself.

class CountWordsAtUrl(Job):
    def run(url=None):
        resp = requests.get(url)

Create example configuration module:

from shire.config import Config as ShireConfig
conf = ShireConfig()

Enqueue job:

from my_jobs import CountWordsAtUrl
from shire_conf import conf
CountWordsAtUrl.delay(conf, pool='default', kwargs={'url': ''})

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shire-0.2.1.tar.gz (20.8 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page