taskue 0.3

Multi stages task queue

TasKue

Multi stages task queue uses Redis as a backend.

Features

• Multi stages.
• Retry on fail for a defined number of retires.
• Capture task logs.
• Task can be taged with label so it will run only on runners that has the same tag.
• Skip task if any task of the previous stages failed.

Installation

pip install taskue

Getting started

Start a runner

taskue runner start --redis-host <localhost> --redis-port <6379>

More options:

• tags: Runner tags, taged tasks will run only on runners which has its tag
• timeout: Runner timeout, task timeout overwrites it (default 1 hour)
• run_untaged_tasks: Allow runner to run untaged tasks (enabled by default).
• path: add the path of a module to include.

How it works

from redis import Redis
from taskue import Taskue, Task 

tskue = Taskue(Redis())

# define tasks
t1 = Task() 
t1.execute(print, "Hello from task 1")                           
t1.retries = 3  # retry up to 3 times on failure                         

t2 = Task()
t2.timeout = 5 # set task timeout to 5
t2.execute(print, "Hello from task 2")  

t3 = Task()
t3.skip_on_failure = True # skip this task if any task of the previous stage failed
t3.execute(print, "Hello from task 3") 

Execute single task

# execute single task
task_uid = tskue.run(t1)

or execute multiple tasks as a workflow

# run task t1 and task t2 in stage 1 and task t3 in stage 2
workflow_uid = tskue.run_workflow([[t1, t2], [t3]])

Hint: All the tasks in the same stage run in parallel, and each stage starts when the previous stage finish.

CLI

taskue --help
Usage: cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  --redis-host TEXT     redis hostname
  --redis-port INTEGER  redis port
  --help                Show this message and exit.

Commands:
  runner
  task
  workflow

commands

• runner:
  • start: start new runner
• task:
  • list: list all tasks
  • get: get task info using its uid
  • logs get task logs
  • delete delete task
• workflow
  • list: list all workflows
  • get: get workflow info using its uid
  • logs get workflow logs
  • delete delete workflow

To Do

• Detect dead runner and reschedule its task.
• Add more docs and examples
• Add tests