job-farmer 0.0.2

A simple job farmer for HPC platforms

Job Farmer

Description

Job Farmer is a python API designed to prepare jobs for HPC platforms supporting SLURM. It collects the job configurations, the associated tasks and prepare the task list and slurm configuration files based on the system constrains and the required ressources.

Getting started

Installation from PyPi

To install the job farmer from PyPi, use pip:

pip install job-farmer

Install from sources

You can also install job-farmer using the sources. First, clone the repository:

git clone [e-mail]:riffard/job_farmer.git

move to the repository:

cd job_farmer

and install the job-farmer:

pip install .

Usage

In the file test/simple_job.py, we are showing a simple job submission.

First, you import the job-farmer:

import job_farmer

Then you create the System object (NERSC/CORI-Haswell in this example) and the Farmer object

system = job_farmer.CoriHaswell()

farmer = job_farmer.Farmer(system)

The nex step consists on creation the JobConfiguration and add it to the Farmer:

job_configuration = job_farmer.JobConfiguration(name='test_job',
                       account='your_account',
                       queue='queue',
                       duration='xx:xx:xx')

farmer.add_global_job_configuration(job_configuration)

You add the Tasks to the Farmer and you link them to the job_configuration

task = job_farmer.Task(commands=[f'export MY_MESSAGE="Hello World!"',
                                 'echo "${MY_MESSAGE}"'], 
                        n_threads=1)
farmer.add_task(job_configuration.name, task)

And it builds the task list and the slurm configuration file:

farmer.build()

Finally, you can submit the jobs to the batch system using:

farmer.submit(dry_run = False)

(by default, dry_run = True).

Note: By default, the job-farmer logs are not displayed. To display them, you need to import the logging package and set the log level to logging.INFO such as:

logging.basicConfig(level=logging.INFO)

Support

If you are encountering an issue, please open a git issue: here

Contributing

Adding a new system or data center

Authors and acknowledgment

License

GNU Affero General Public License v3.0