Skip to main content

Simple task farmer using file locks to syncrhonize among multiple nodes.

Project description

PyTaskFarmer

This is a small project of a Python TaskFarmer for running jobs at NERSC's Cori (though it should be flexible to run on other systems). It is very loosely based on the concept of Shane Canon's TaskFarmer.

Full Documentation

Usage

The executable script is:

usage: pytaskfarmer.py [-h] [--proc [Processes]] [--timeout TIMEOUT]
                        [--workdir WORKDIR] [--verbose VERB]
                        [--runner RUNNER] [--tasklist TASKLISTHANDLER]
                        tasklist

The tasklist argument is a simple text file with one task per line. The interpretation of the task is up to the TASKLISTHANDLER. By default, the task is treated as a command to run. It is not important how complex the command is.

The --verbose flag adds a bit more output (though not much) telling you what the script is doing as it runs (default False).

The --timeout option allows you to set a timeout for the script, so that after some number of seconds the tasks will be automatically killed (default none).

The --proc option tells the script how many parallel workers to create (default8).

The --workdir option tells the script where to store the progress (task status, log files..) of a single run (default is tasklist_workdir).

The --runner options indicates which runner to execute the command with. See the dedicated section on the available runners and how they work.

The --tasklist options indicates which tasklist handler to parse the tasklist with. See the dedicated section on the available runners and how they work.

What it does (60 second version)

The basic behavior, with the default runner/handler, is as follows. Each access to a file is protected using a file locking mechanism.

  1. The tasklist is read and a toprocess file is created in the workdir with unprocessed tasks.

  2. A number of workers (multiprocessing.Pool) are constructed to run on the tasks.

  3. When some work is done, the command is placed into a finished or failed files, depending on the status code.

  4. Duration and start times of completed tasks (timeline) are saved into a timeline.json file. This can then be opened with Perfetto.

  5. The tasks are processed by the workers until 1) the work is completed; 2) the timeout is reached; or 3) a signal is intercepted.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

PyTaskFarmer-0.1.0.tar.gz (12.0 kB view details)

Uploaded Source

Built Distribution

PyTaskFarmer-0.1.0-py3-none-any.whl (13.9 kB view details)

Uploaded Python 3

File details

Details for the file PyTaskFarmer-0.1.0.tar.gz.

File metadata

  • Download URL: PyTaskFarmer-0.1.0.tar.gz
  • Upload date:
  • Size: 12.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9

File hashes

Hashes for PyTaskFarmer-0.1.0.tar.gz
Algorithm Hash digest
SHA256 a52c211cc0c3e0e00a89c6230e6cf7e574501a4fe7421a4adf1b19d09a8a28dd
MD5 642e7846e54d00c2287231efa8ffe4f0
BLAKE2b-256 befe08dff34f40340abb406fa81365fd00938aa35ddb059fcde237c4027eb6f4

See more details on using hashes here.

File details

Details for the file PyTaskFarmer-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: PyTaskFarmer-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 13.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.9

File hashes

Hashes for PyTaskFarmer-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 f865e1412ec5326ef4c0cd9911cacc842331e10fa5cc0e1719ae9a5a17f05e6f
MD5 6c1439ca36d38a069844efcbc4120a2f
BLAKE2b-256 6ebdad88b1105008e9ad3b8f39095e85202db27a765d26d22963c97e069b293f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page