taskwatcher 1.7

Package taskwatcher

taskwatcher

DESCRIPTION

'taskwatcher' is a set of libraries/commands to turn scripts/programs into tasks :

• launch.py : Starts a new task under taskwatcher monitoring
• control.py : General control commands
• parse.py : Feedback file parser, returns a json string from the feedback file

An history of the terminated tasks is kept.
A status of the currently running tasks is available.

Tasks are monitored by checking if pid is still in process list. Optionally, if feedback is provided by the launched program, program is expected to provide heartbeat by updating the feedback file.

An optional task timeout may terminated the task if no hearbeat was received during the allowed 'timeout' period.

There are conditions applies so a task can start, multiple scenario :

1. Task does not provide feedack, can run multiple time:

• no control need, a call to lauch would be enough, task would be automatically reserved and launched

2. Task does not provide feeback but only one of this kind should work at a given timer:

• a reservation with a task name and option --unique set should be done befor calling launch. A taskid would be returned, use this task id in call to lauch.

3. Task provides feeback.

• Reservation is required prior calling launch.
• If task should only run once, use --unique option during reservation.

Prior to any launch call to start a new task, a taskid (number) should have been reserved with a call control.py --reserve.

A feedback file is a simple text file with key/value pairs. It can be parsed with parse.py to return a json formated string.

Limitations

• Currently only supports mono-process program (track a single pid)

Design

Choice was made to avoid complex sockets operations. Simple design with feedback files also allows simple state recovery, simplicity and versatility of the feedback parameters. A basic file-based sqlite database is used to store running tasks data and keep track of historical tasks, instead of implementing a daemon.

Configuration

• Currently no configuration file is implemented. All options should be provided using cli options.
• No centralized daemon:
  • a launcher is started with the program to run
  • getting feeback from the running task can be done by calling control.py with command 'feedback'
  • history and current status of running tasks are located in sqlite db

Expectation from the launched command

A command may be ran without specific requirement, however to benefit from additional features, the run command may provide feedback via a text file.
The suggestion is to add to programs launched with taskwatcher a command line option (--feedback) to generate the feedback file.
The feedback file is expected to be named based on the taskid : feedback_TASKID.log Example :

• Call without taskwatcher :
  checkitbaby.py --playbook myPlaybook --playlist myPlaylist --run 1 --dryrun

• Same call with taskwatcher :

  • Get a taskid : control.py --reserve ==> Got 1
  • Launch task using taskid 1 :
    launch.py --taskid 1 --name 'Runner' --info1 'Info1' --info2 'Info2' --info3 'Info3' --feedpath /fortipoc/playbooks/myPlaybook/run/1 --db /fortipoc/taskwatch/sqlite.db --timeout 30 -- checkitbaby.py --playbook myPlaybook --playlist myPlaylist --run 1 --dryrun --feedback feedback_1.log

  Notes :

  • info1, info2, info3 are optional, it can be usefull to store side information in the task manager
  • the command to run is located after the --
  • called program is informed with --feedback feedback_1.log that it should write a feedback file named feedback_1.log

Running task status

• without feedback :

  • running : pid of the task is seen

• with feedback :

  • running : pid of the task is seen,
    last feedback file update was done in less than 25% of timeout timer

  • silent : pid of the task is seen,
    last feedback file update was done in less than 50% of timeout timer

  • stalled : pid of the task is seen,
    last feedback file update was done in less than 75% of timeout timer

Feedback file syntax

The feedback file is named 'feedback.log', it should be generated by the launched program. Any kind of usefull information could be delivered as long as :

• it is key/value pair
• line starts with a keywork emcompassed with [] to specify the information keyword.
• keyword should have no spaces and should not start with a digit
• value provided should immediately follow the [keyword] without spaces
• line starting with # are considered comments/debug and will be ignored
• empty lignes will be ignored
• a keyword without any value on a line clears the keyword and value information from the feedback
• [] (without keyword) clears all key/value pairs collected up to this point. Note : this could be usefull to end the task with a clear followed by a report.

Feedback file processing

• launch.py does not parse feedback file. It only checks the file update from the file update time for the task timeout fonction.

• control.py processed feedback file to provide output.
  The last read value for a keyword updates any precedent values.

• use parse.py to parse and retrieve json from the feedback file.

• Example of a feedback.log

[info]Starting test
[playbook]myPlaybook
[info]
[heartbeat]
[testcase_id]001
[testcase_name]Initialization
[progress]12
[progress]15
[testcase_id]002
[testcase_name]Setting up topology
[progress]10

• If processed until this point, the above output would provide information like:

{ 
  "playbook"      : "myPlaybook",
  "testcase_id"   : "002",
  "testcase_name" : "Setting up topology",
  "progress"      : "10"
}

Notes :

• 'info' is not provided because it was cleared by the '[info]' line in line 3
• 'heartbeat' has no information, it would only reset the 'timeout' counter

launch.py

Roles : 
   - launches command
   - monitor the command checking its pid in process list
   - monitor activity of the feedback file (by its update timing information)
   - kills command if not updating feedback file within the timeout 
   - update the running task db about process state, duration and timer status
   - manage task termination : archive task on database, delete feedback file

Pre-requisite :
A unique taskid should have been reserved from a call to control.py to avoid duplicates.
If no reservation was made, the task won't start

Usage : launch.py --db <database> --taskid <taskid> --command '<process or script with all its options>'

Parameters :
--db       <database>     : sqlite database file
--taskid   <taskid>       : task identifier (could be a number or a generated random string (8 chars max)    

Optional parameters :  
--name     <name>         : Name for the task. Use command name if not provided
--info1 INFO1             : any information (optional)
--info2 INFO2             : any information (optional)
--info3 INFO3             : any information (optional)

--feedpath <path>         : Feedback path where feedback.log is expected
--timeout  <seconds>      : a value in second after which the command is considered timeout
				            and should be kill (any update in feedback.log resets the timer)

control.py

Roles :
   - Provide list of all running tasks with their latests status
   - Provide history of terminated tasks
   - Provide all latests feedback information from the task
   - Manage history
   - Kill task (and cleanup feedback file if necessary)
   - Initialise (or re-initialize db)
   - Set a reservation for a task id

Usage : control.py --db <database> <command>

Options :
--db <database>      : sqlite database file

List of available commands :

--initialize         : Create or recreates a task database (all info is lost)

--update             : Update database time informations (task duration)

--list               : Provides a table displaying the list of the currently running tasks with : 
                       [ taskid, name, info1, info2, info3, pid, status, starttime, duration(s), feedback(yes/no), timer(s), timeout(s) ]

--reserve            : Returns a unique taskid, reserved for future call of the launcher

--feedback <taskid>  : Returns a json formatted output of the feedback values for the given task
                     : Only available if the command provides feedback (feedback=yes in list)

--history            : Dump all historical tasks completed
--clear              : Clear all tasks history

--kill <taskid>      : Request to terminate a specific task
--killall <taskname> : Request to terminate all tasks named <taskname>

parse.py

usage: parse.py [-h] [--debug] --feedback filename

Task controller

optional arguments:
  -h, --help           show this help message and exit
  --debug, -d          turn on debug
  --feedback filename  selects feedback file to process


Example: 
 ./parse.py --feedback tests/textfile_progress.txt
 {\"progress\": \"100\"}

sqlite database

An sqlite database is used for 3 purposes :

• keep track of the current running tasks :
  Launcher.py updates tasks status but does not process feedback

• keep track of the latest feedback from the launched command
  control.py called with --feedback parses command feedback file, stores information and returns a json object

• keep an history of the previously completed tasks
  control.py called with --history

Table format

* Table tasks:
  Keeps track of running tasks status
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------
  |       id(#1)        |  name | info1 | info2 | info3 |   pid   |   status   |   feedback    |  reservetime |   starttime   | duration |  lastupdate  | timeout |
  | INTEGER PRIMARY KEY |  TEXT | TEXT  | TEXT  | TEXT  | INTEGER |  TEXT(#2)  |  INTEGER(#3)  |  INTEGER(#4) |   INTEGER(#4) | INTEGER  |  INTEGER(#4) | INTEGER |
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------

  Note : 
    #1 : should be automatic (use null during insert)
    #2 : RESERVED|RUNNING|SILENT|STALLED
    #3 : 0 if no feedback provided ; 1 if feedback provided
    #4 : unix date format

  taskid reservation consists of inserting a new task with all field empty, except status=RESERVED and reservetime set

* Table feedbacks:
  Keeps track of data feebacks from the run command, stored as json key/value pairs
  ---------------------------------------
  |    id   | feedback  |   lastupdate  |
  | INTEGER |  BLOB(#1) |   INTEGER(#2) |
  ---------------------------------------

  Note :
  #1 : json format expected
  #2 : unix date format


* Table history:
  Keeps track of the completed tasks
  Final state of json feedback is stored (this allows to store json reports before the command terminates)
  --------------------------------------------------------------------------------------------------------------------------------------------------------
  |       id(#0)        |  taskid(#1)  | taskname |  info1 | info2 | info3 | termsignal | termerror |   starttime   |   endtime   | duration | feedback  | 
  | INTEGER PRIMARY KEY |   INTEGER    |   TEXT   |  TEXT  | TEXT  | TEXT  |  TEXT(#1)  |  TEXT(#2) |   INTEGER(#3) | INTEGER(#3) | INTEGER  |  BLOB(#4) |
  --------------------------------------------------------------------------------------------------------------------------------------------------------

  Notes :
  #1 : keeps track of the type of termination signal
  #2 : keeps track of the terminaison error message if any
  #3 : unix date format
  #4 : json format expected