Simple task farmer using file locks to syncrhonize among multiple nodes.
Project description
PyTaskFarmer
This is a small project of a Python TaskFarmer for running jobs at NERSC's Cori (though it should be flexible to run on other systems). It is very loosely based on the concept of Shane Canon's TaskFarmer.
Usage
The executable script is:
usage: pytaskfarmer.py [-h] [--proc [Processes]] [--timeout TIMEOUT]
[--workdir WORKDIR] [--verbose VERB]
[--runner RUNNER] [--tasklist TASKLISTHANDLER]
tasklist
The tasklist
argument is a simple text file with one task per line. The
interpretation of the task is up to the TASKLISTHANDLER
. By default, the task
is treated as a command to run. It is not important how complex the command is.
The --verbose
flag adds a bit more output (though not much) telling you what
the script is doing as it runs (default False).
The --timeout
option allows you to set a timeout for the script, so that after
some number of seconds the tasks will be automatically killed (default none).
The --proc
option tells the script how many parallel workers to create
(default8).
The --workdir
option tells the script where to store the progress (task
status, log files..) of a single run (default is tasklist_workdir
).
The --runner
options indicates which runner to execute the command with. See
the dedicated section on the available runners and how they work.
The --tasklist
options indicates which tasklist handler to parse the tasklist
with. See the dedicated section on the available runners and how they work.
What it does (60 second version)
The basic behavior, with the default runner/handler, is as follows. Each access to a file is protected using a file locking mechanism.
-
The tasklist is read and a
toprocess
file is created in theworkdir
with unprocessed tasks. -
A number of workers (
multiprocessing.Pool
) are constructed to run on the tasks. -
When some work is done, the command is placed into a
finished
orfailed
files, depending on the status code. -
Duration and start times of completed tasks (timeline) are saved into a
timeline.json
file. This can then be opened with Perfetto. -
The tasks are processed by the workers until 1) the work is completed; 2) the timeout is reached; or 3) a signal is intercepted.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for PyTaskFarmer-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f865e1412ec5326ef4c0cd9911cacc842331e10fa5cc0e1719ae9a5a17f05e6f |
|
MD5 | 6c1439ca36d38a069844efcbc4120a2f |
|
BLAKE2b-256 | 6ebdad88b1105008e9ad3b8f39095e85202db27a765d26d22963c97e069b293f |