tc4v-tasq 0.1.0

A task queue for parallel processing

tasq.py

A parallel safe task queue on disk.

About

tasq.py maintains a database of tasks and let you claim and mark tasks as done in a multi-process safe way. It has two backends, a sqlite based one for general single machine multi-process use, and a file based one for multi-machine multi-process use (on a shared filesystem).

Usage

A task is only defined as a string of text, or name, that should be unique. The interpretation of the name is up to the user.

To create a new task queue, use init and provide the first list of task from a file or stdin:

$ tasq init tasks.db - <<EOF
./work/task1 3
./work/task2 9

# comments and empty lines are ignored
./work/task3 0 # inline comments work too
EOF

Tasks can be added later in a similar fashin with add:

$ tasq add tasks.db - <<EOF
./work/task4 blublublu
./work/task5 "it uses shlex so spaces are ok"
./work/task6 blablabla
EOF

To see the current state of the queue, use list:

$ tasq list tasks.db
a6cc64a4401b4e0a94489b43bd22ac72 pending 2025-11-22 21:53:31 ['./work/task1', '3']
390dda92b89f4b4a9ed519fc86356fd7 pending 2025-11-22 21:53:31 ['./work/task2', '9']
a2380735393e4f07a7b4b67eb7a969eb pending 2025-11-22 21:53:31 ['./work/task3', '0']
482955a8421940a5b4874d4987affb30 pending 2025-11-22 21:53:37 ['./work/task4', 'blublublu']
1d875235a5a64d16921fa9d5e0d54309 pending 2025-11-22 21:53:37 ['./work/task5', 'it uses shlex so spaces ar
e ok']
b030397203a94303b1a6a1960ba78aaf pending 2025-11-22 21:53:37 ['./work/task6', 'blablabla']

The hex string at the begining is a unique identifier for the task that is used to interact with it.

Tasks can be claimed using claim:

$ tasq claim tasks.db -n 2 --data
390dda92b89f4b4a9ed519fc86356fd7 ["./work/task2", "9"]
a6cc64a4401b4e0a94489b43bd22ac72 ["./work/task1", "3"]

and marked done with done:

$ tasq done tasks.db a6cc64a4401b4e0a94489b43bd22ac72

See also tasq help for more commands.

There is also a second script pll that does something similar to GNU parallel but for a task queue.

For example the following command:

$ pll tasks.db -n 2 ./process.sh '{0}' '{1}'

will process all tasks calling the command ./process.sh with the task arguments (placeholder follow str.format syntax) using two parallel workers. The outputs will be printed concatenates with # <TASK_ID> line above each outputs. Because it uses the task queue, you can have more than one pll running in parallel (for example if using slurm, each slurm job can use pll).