fenpei 2.7.2

Distribution of tasks.

This little tool helps in scheduling, tracking and aggregating calculations and their results. It forms the step that brings you from ‘a directory with working code for a job’ to ‘running dozens of jobs and getting results easily’.

pip install fenpei

This is intended to be used to run multiple intensive computations on a (linux) cluster. At present, it assumes a shared file system on the cluster.

It takes a bit of work to integrate with your situation but it is very flexible and should make your life easier after setting it up. Some features:

• Jobs are created in Python files, making it short and extremely flexible.

• It uses a command line interface (some shell experience required) to easily start, stop or monitor jobs.

• Easy to use with existing code and easily reproducible, since it works by creating isolated job directories.

• Can replaces scheduling queue functionality and start jobs through ssh, or can work with existing systems (slurm and qsum included, others implementable).

• Flexibility for caching, preparation and result extraction.

• Uses multi-processing and can easily use caching for greater performance, and symlinks to save space.

Note that:

• You will have to write Python code for your specific job, as well as any analysis or visualization for the extracted data.

• Except for status monitoring mode, it derives the state on each run, it doesn’t keep a database that can get outdated or corrupted.

One example to run reproducible jobs with Fenpei (there are many ways):

• Make a script that runs your code from source to completion for one set of parameters.

• Subclass the ShJobSingle job and add all the files that you need in get_nosub_files.

• Replace all the parameters in the run script and other config files by {{ some_param_name }}. Add these files to get_sub_files.

• Make a Python file (example below) for each analysis you want to run, and fill in all the some_param_name with the appropriate values.

• From a shell, use python your_jobfile.py -s to see the status, then use other flags for more functionality (see below).

• Implement is_complete and result in your job (and crash_reason if you want -t) (others can be overridden too, if you require special behaviour).

• Add analysis code to your job file if you want to visualize the results.

Example file to generate jobs:

def generate_jobs():
    for alpha in [0.01, 0.10, 1.00]:
        for beta in range(0, 41):
            dict(name='a{0:.2f}_b{1:d}'.format(alpha, beta), subs=dict(
                alpha=alpha,
                beta=beta,
                gamma=5,
                delta='yes'
            ), use_symlink=True)

def analyze(queue):
    results = queue.compare_results(('J', 'init_vib', 'init_rot',))
    # You now have the results for all jobs, indexed by the above three parameters.
    # Visualization is up to you, and will be run when the user adds -x

if __name__ == '__main__':
    jobs = create_jobs(JobCls=ShefJob, generator=generate_jobs(), default_batch=splitext(basename(__file__))[0])
    queue = SlurmQueue(partition='example', jobs=jobs, summary_func=analyze)
    queue.run_argv()

This file registers many jobs for combinations of alpha and beta parameters. You can now use the command line:

usage: results.py [-h] [-v] [-f] [-e] [-a] [-d] [-l] [-p] [-c] [-w WEIGHT]
                  [-q LIMIT] [-k] [-r] [-g] [-s] [-m] [-x] [-t] [-j]
                  [--jobs JOBS] [--cmd ACTIONS]

distribute jobs over available nodes

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         more information (can be used multiple times, -vv)
  -f, --force           force certain mistake-sensitive steps instead of
                        failing with a warning
  -e, --restart         with this, start and cleanup ignore complete
                        (/running) jobs
  -a, --availability    list all available nodes and their load (cache reload)
  -d, --distribute      distribute the jobs over available nodes
  -l, --list            show a list of added jobs
  -p, --prepare         prepare all the jobs
  -c, --calc            start calculating one jobs, or see -z/-w/-q
  -w WEIGHT, --weight WEIGHT
                        -c will start jobs with total WEIGHT running
  -q LIMIT, --limit LIMIT
                        -c will add jobs until a total LIMIT running
  -k, --kill            terminate the calculation of all the running jobs
  -r, --remove          clean up all the job files
  -g, --fix             fix jobs, check cache etc (e.g. after update)
  -s, --status          show job status
  -m, --monitor         show job status every few seconds
  -x, --result          run analysis code to summarize results
  -t, --whyfail         print a list of failed jobs with the reason why they
                        failed
  -j, --serial          job commands (start, fix, etc) may NOT be run in
                        parallel (parallel is faster but order of jobs and
                        output is inconsistent)
  --jobs JOBS           specify by name the jobs to (re)start, separated by
                        whitespace
  --cmd ACTIONS         run a shell command in the directories of each job
                        that has a dir ($NAME/$BATCH/$STATUS if --s)

actions are executed (largely) in the order they are supplied; some actions
may call others where necessary

Pull requests, extra documentation and bug reports are welcome! It’s Revised BSD-licensed so you can do many things.