tq1 0.4.6

Simple Command Line Job Manager

TQ -- Simple Command Line Job Manager

Description in English:

TQ (Task Queue) is a simple Command Line Job Manager. In principle TQ is a very flexible and smart atd(8), which could arrange a series of jobs in an efficient way.

(1) By default TQ will run the jobs one by one in the FIFO order.

(2) A job with high priority will be processed earlier.

(3) Given the estimated occupancy coefficient, jobs can be executed in parallel as long as possible.

The management of job queue is based on SQLite3 database, in which information about every job, including the start and end time, is stored.

中文描述(Description in Chinese):

在一位大佬的怂恿下，我把自己炼丹用的队列管理器tq传到了pypi上。 tq本质上就是一个异常灵活的atd，可以用来安排执行一系列耗时的命令。 默认情况下tq会串行（FIFO）执行给定的任务队列。如果指定了优先级 或者预计资源占用比，tq还能根据情况优先执行高优先任务，或者并行 执行若干非独占的任务。队列的管理依赖于SQLite3数据库，每一个任务 的各类信息，包括起止时间，都会被记录在其中。

Features

1. TQ keeps your command in a queue and execute them one by one.

2. Priority attribute is supported. The scheduler will execute high priority command lines first.

3. Resource occupancy coefficient is supported. The scheduler will execute command lines in parallel as long as possible.

4. The program output will be redirected to tq.out and tq.err under the directory where the task was added.

5. The queue is saved in SQLite3. Won't lose any data even if a powerloss had happend.

6. Dependency free and light weight. This tool only depends on python3 itself. (with SQLite3 support)

Install

This tools is available on Pypi. Just issue the following command:

pip3 install tq1

Note that python3 >= 3.6 is required due to new language features used in the code. There is no plan to support lower version of python.

Real-Life Example

TQ can be used to deal with some commands in an async manner. e.g.

$ tq -- git push  # Doesn't block. Have it done in async.
$ vim mycode.py

TQ can be used to manage a series of computation experiments, such as deep learning experiments, e.g.

$ tq r5 -- caffe train -solver net1forfun.prototxt
$ tq r5 -- caffe train -solver net2forfun.prototxt
$ tq 1 5 -- python3 train.py --lr 1e-2
$ tq 1 5 -- python3 train.py --lr 1e-3
$ tq 1 5 -- python3 train.py --lr 1e-4
$ tq 1 5 -- python3 train.py --lr 1e-5
$ tq p10 -- python3 important_train.py

One can just put many computation tasks in the queue, and TQ will smartly schedule these experiments according to the given priority and resource occupancy parameters.

Usage

Usage: tq ACTION [COMMAND_ARGS]
       tq [P R] -- TASK

Available Actions:
    start      start TQ's daemon
    stop       stop TQ's daemon
    log        dump log to screen
    ls         fancy print of task queue
    db         print database content to screen
    rm <ID>    remove task with specified id, see ID with tq ls
    clean      remove finished tasks from queue
    purge      remove log file and sqlite3 db file

Apending Task:
    -- TASK        append TASK to the queue
    p<P> -- TASK   append TASK with priority P to the queue
    r<R> -- TASK   append TASK with resource occupancy R to the queue
    P R -- TASK    append TASK with priority P and estimated occupancy R
                   int P default  0 range [INT_MIN, INT_MAX], large=important
                   int R detault 10 range [1,       10],      large=consuming

Usage Examples

0. Daemon: to start or stop the daemon
     tq start
     tq stop
1. Serial: the two given tasks should be executed one by one
     tq -- sleep 100
     tq -- sleep 100
2. Parallel: each task occupies 40% of resource.
   In this example two tasks will be active at the same time.
     tq r4 -- sleep 100
     tq r4 -- sleep 100
     tq r4 -- sleep 100
3. Priority: break the FIFO order of tasks. 1 > default Priority.
     tq p1 -- sleep 100
4. Special Case: run the given task right away ignoring Pri and Rsc
     tq 1 0 -- sleep 100

See Also

atd(8), HTCondor, Torque, Slurm, PBS.

https://wiki.debian.org/HighPerformanceComputing

License

MIT License.