This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

This little tool helps in scheduling, tracking and aggregating calculations and their results. It forms the step that brings you from ‘a directory with working code for a job’ to ‘running dozens of jobs and getting results easily’.

pip install fenpei

This is intended to be used to run multiple intensive computations on a (linux) cluster. At present, it assumes a shared file system on the cluster.

It takes a bit of work to integrate with your situation but it is very flexible and should make your life easier after setting it up. Some features:

  • Jobs are created in Python files, making it short and extremely flexible.
  • It uses a command line interface (some shell experience required) to easily start, stop or monitor jobs.
  • Easy to use with existing code and easily reproducible, since it works by creating isolated job directories.
  • Can replaces scheduling queue functionality and start jobs through ssh, or can work with existing systems (slurm and qsum included, others implementable).
  • Flexibility for caching, preparation and result extraction.
  • Uses multi-processing and can easily use caching for greater performance, and symlinks to save space.

Note that:

  • You will have to write Python code for your specific job, as well as any analysis or visualization for the extracted data.
  • Except for status monitoring mode, it derives the state on each run, it doesn’t keep a database that can get outdated or corrupted.

One example to run reproducible jobs with Fenpei (there are many ways):

  • Make a script that runs your code from source to completion for one set of parameters.
  • Subclass the ShJobSingle job and add all the files that you need in get_nosub_files.
  • Replace all the parameters in the run script and other config files by {{ some_param_name }}. Add these files to get_sub_files.
  • Make a Python file (example below) for each analysis you want to run, and fill in all the some_param_name with the appropriate values.
  • From a shell, use python your_jobfile.py -s to see the status, then use other flags for more functionality (see below).
  • Implement is_complete and result in your job (and crash_reason if you want -t) (others can be overridden too, if you require special behaviour).
  • Add analysis code to your job file if you want to visualize the results.

Example file to generate jobs:

def generate_jobs():
    for alpha in [0.01, 0.10, 1.00]:
        for beta in range(0, 41):
            dict(name='a{0:.2f}_b{1:d}'.format(alpha, beta), subs=dict(
                alpha=alpha,
                beta=beta,
                gamma=5,
                delta='yes'
            ), use_symlink=True)

def analyze(queue):
    results = queue.compare_results(('J', 'init_vib', 'init_rot',))
    # You now have the results for all jobs, indexed by the above three parameters.
    # Visualization is up to you, and will be run when the user adds -x

if __name__ == '__main__':
    jobs = create_jobs(JobCls=ShefJob, generator=generate_jobs(), default_batch=splitext(basename(__file__))[0])
    queue = SlurmQueue(partition='example', jobs=jobs, summary_func=analyze)
    queue.run_argv()

This file registers many jobs for combinations of alpha and beta parameters. You can now use the command line:

usage: results.py [-h] [-v] [-f] [-e] [-a] [-d] [-l] [-p] [-c] [-w WEIGHT]
                  [-q LIMIT] [-k] [-r] [-g] [-s] [-m] [-x] [-t] [-j]
                  [--jobs JOBS] [--cmd ACTIONS]

distribute jobs over available nodes

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         more information (can be used multiple times, -vv)
  -f, --force           force certain mistake-sensitive steps instead of
                        failing with a warning
  -e, --restart         with this, start and cleanup ignore complete
                        (/running) jobs
  -a, --availability    list all available nodes and their load (cache reload)
  -d, --distribute      distribute the jobs over available nodes
  -l, --list            show a list of added jobs
  -p, --prepare         prepare all the jobs
  -c, --calc            start calculating one jobs, or see -z/-w/-q
  -w WEIGHT, --weight WEIGHT
                        -c will start jobs with total WEIGHT running
  -q LIMIT, --limit LIMIT
                        -c will add jobs until a total LIMIT running
  -k, --kill            terminate the calculation of all the running jobs
  -r, --remove          clean up all the job files
  -g, --fix             fix jobs, check cache etc (e.g. after update)
  -s, --status          show job status
  -m, --monitor         show job status every few seconds
  -x, --result          run analysis code to summarize results
  -t, --whyfail         print a list of failed jobs with the reason why they
                        failed
  -j, --serial          job commands (start, fix, etc) may NOT be run in
                        parallel (parallel is faster but order of jobs and
                        output is inconsistent)
  --jobs JOBS           specify by name the jobs to (re)start, separated by
                        whitespace
  --cmd ACTIONS         run a shell command in the directories of each job
                        that has a dir ($NAME/$BATCH/$STATUS if --s)

actions are executed (largely) in the order they are supplied; some actions
may call others where necessary

Pull requests, extra documentation and bug reports are welcome! It’s Revised BSD-licensed so you can do many things.

Release History

Release History

2.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
fenpei-2.3-py2.py3-none-any.whl (37.2 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Nov 17, 2016
fenpei-2.3.tar.gz (28.2 kB) Copy SHA256 Checksum SHA256 Source Nov 17, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting