Skip to main content

Execute a certain sequence of tasks and later their cleanups in reverse order.

Project description

# TaskRunner

Execute a certain sequence of tasks and later their cleanups. It is useful for
running tasks with many varying configurations. It doesn't have any
dependencies, just the standard library.

# file examples/
import taskrunner
class ExampleTask(taskrunner.Task):
def __init__(self, msg, clean_msg, **kwargs):
super(ExampleTask, self).__init__(**kwargs)
self.msg = msg
self.clean_msg = clean_msg

def run(self, context):
print self.msg

def cleanup(self, context):
print self.clean_msg

task1 = {'task': ExampleTask,
'name': 'task1',
'msg': 'hello world',
'clean_msg': 'goodbye'}
task2 = {'task': ExampleTask,
'name': 'task2',
'msg': 'hello again',
'clean_msg': ':)'}

pipeline = [task1, task2]

$ bin/taskrunner -f examples/ pipeline
2013-12-02 19:46:37,952 - taskrunner - INFO - =========== run task1 ===========
hello world
2013-12-02 19:46:37,953 - taskrunner - INFO - =========== run task2 ===========
hello again
2013-12-02 19:46:37,953 - taskrunner - INFO - --------- cleanup task2 ---------
2013-12-02 19:46:37,953 - taskrunner - INFO - --------- cleanup task1 ---------

### How it works

The pipeline is a list of task configurations, which are normal Python
dictionaries with the special item `'task':ExampleTask`. It goes trough the
list and for each task, it instantiates `ExampleTask` with the rest of the
dictionary content as parameters. Then it executes `` for all
of the tasks. After it passes trough the whole list, it goes trough it in
reverse order and executes `ExampleTask.cleanup()` for each item. The tasks can
write into `context` and the content of it will be passed to the next task.

### Usage

You can specify the pipeline directly as arguments:

$ bin/taskrunner -f examples/ task1 task2

Or you can combine multiple pipelines, which will run all the tasks from each

$ bin/taskrunner -f examples/ pipeline another_task_pipeline

Or even combine pipelines and tasks (this will run *task2* twice):

$ bin/taskrunner -f examples/ pipeline task2

To use the tool as a library, you can directly use `execute`:

taskrunner.execute([task1, task2])

#### Taking control of the cleanup execution

Sometimes you want to only execute the `run()` part of the tasks, debug
something and only run the cleanups after you are done. To skip the cleanups,
you can do:

$ bin/taskrunner -f examples/ pipeline --cleanup=never

To run the cleanups only:

$ bin/taskrunner -f examples/ pipeline --cleanup=pronto

You can also use the options `--cleanup=on_success` or `--cleanup=on_failure`,
which will get executed based on how the `run` turned out.

Don't forget to make the cleanups independent of the runs, otherwise this won't

#### Exception and signal handling

When an exception occurs in the task run, its traceback is logged and it jumps
right into the cleanups. However, it doesn't clean up the tasks that didn't run
(but it does clean up the task which failed and got only partially executed).

If you get an exception during some cleanup, the traceback is logged but
execution continues with the next task's cleanup.

The list of errors gets logged again after everything else finishes, in the
order they happened.

If you terminate the run using `ctrl-c` (also known as *SIGINT*), it will go
straight to the cleanups. Sending the termination signal again will stop it
completely. This works for the *SIGTERM* signal too.

#### The name of a task

By default, the name of a task is the class name. To have more readable logs,
you can specify the keyword `name` in the task configuration. The task names
can be important for configuration redefinition from the command line.

#### Redefining the task configuration trough CLI arguments

Sometimes you want to run a sequence of tasks with some changes in their
configuration, but don't want to change the files. You can redefine it using
the parameter `-D`.

$ bin/taskrunner -f examples/ pipeline -D task1.msg=ping

It can't contain any spaces, has to be in the exact format of
`taskname.key1.key2.key3...=newvalue`, where `taskname` is either the name of
the task specified in the configuration dictionary or the class name. If more
tasks have the same name, it will get rewritten for all of them. For example,

$ bin/taskrunner -f examples/ pipeline -D ExampleTask.msg=ping

will change the message for both `task1` and `task2`, because they have the
same class name.

#### Using multiple files for the task configurations

You probably don't want to have everything in a single file. You can load
multiple modules and reference the tasks normally.

$ bin/taskrunner -f examples/ -f examples/ \
mytask task1 task2

In case you have any name conflicts, you can specify the name of the module.

$ bin/taskrunner -f examples/ -f examples/ \

### Best practices for writing tasks and their configurations
* don't make the `cleanup` method dependent on `run`, because with the
option `--cleanup=pronto`, the `run` method won't get executed
* don't assume that the `run` got executed completely
* put the tasks into a separate file, which will be imported in the file with
the task configurations
* use the minimum of Python features in the task configuration files (which are
just `.py` files), variable definitions and if conditions are usually
sufficient. You will be later able to switch to some other configuration

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

taskrunner-0.2.1.tar.gz (13.1 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page