Skip to main content

Heavy-job queue processor

Project description

Heavy-job queue processor

Installation

pip install shire

Create config file

Run shire-cli create_config, follow instructions.

An example of the shire.cfg:

[connection]
redis_url = redis://127.0.0.1:6379/0
db_url = postgres://peewee:123@127.0.0.1:5432/peewee

[hostler]
check_time = 5

[shire]
venv_path = /path/to/your/virtualenv
sys_path = /path/to/your/project:/path/to/external/library
host = default

[scribe]
per_pool = 1
log_max_size = 104857600
log_directory = logs
max_logs = 5
default_log = default.log

[pool]
check_time = 30

[whip]
limits_update_time = 60
check_time = 1

Queue management

Interactive shire config creation
shire-cli -c /path/to/your/shire.cfg create_config
Run whip (Enqueue jobs to pools)
shire-cli -c /path/to/your/shire.cfg run_whip
Run scribe (Shire job logs writer)
shire-cli -c /path/to/your/shire.cfg run_scribe
Run hostler (Shire failed jobs restarter)
shire-cli -c /path/to/your/shire.cfg run_hostler
Run pool (Shire job executor)
shire-cli -c /path/to/your/shire.cfg run_pool –name=pool_name
Run multiple pools with one master-process
shire-cli -c /path/to/your/shire.cfg start_pools –names=pool_name,another_pool

Example

Run shire
shire-cli -c /path/to/your/shire.cfg run_whip shire-cli -c /path/to/your/shire.cfg run_pool –name=default

Create example jobs module:

# my_jobs.py
from shire.job import Job
import requests

def save(text_len):
    # save - your external function to save result data. Shire not save any results itself.
    pass

class CountWordsAtUrl(Job):
    def run(url=None):
        resp = requests.get(url)
        save(len(resp.text.split()))

Create example configuration module:

# shire_conf.py
from shire.config import Config as ShireConfig
conf = ShireConfig()
conf.load('/path/to/your/shire.cfg')

Enqueue job:

from my_jobs import CountWordsAtUrl
from shire_conf import conf
CountWordsAtUrl.delay(conf, pool='default', kwargs={'url': 'https://github.com/suguby/shire'})

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for shire, version 0.2.1
Filename, size File type Python version Upload date Hashes
Filename, size shire-0.2.1.tar.gz (20.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page