A simple job farmer for HPC platforms
Project description
Job Farmer
Description
Job Farmer is a python API designed to prepare jobs for HPC platforms supporting SLURM. It collects the job configurations, the associated tasks and prepare the task list and slurm configuration files based on the system constrains and the required ressources.
Getting started
Installation from PyPi
To install the job farmer from PyPi, use pip:
pip install job-farmer
Install from sources
You can also install job-farmer using the sources. First, clone the repository:
git clone git@gitlab.com:riffard/job_farmer.git
move to the repository:
cd job_farmer
and install the job-farmer:
pip install .
Usage
In the file test/simple_job.py
, we are showing a simple job submission.
First, you import the job-farmer:
import job_farmer
Then you create the System
object (NERSC/CORI-Haswell in this example) and the Farmer
object
system = job_farmer.CoriHaswell()
farmer = job_farmer.Farmer(system)
The nex step consists on creation the JobConfiguration
and add it to the Farmer
:
job_configuration = job_farmer.JobConfiguration(name='test_job',
account='your_account',
queue='queue',
duration='xx:xx:xx')
farmer.add_global_job_configuration(job_configuration)
You add the Tasks
to the Farmer
and you link them to the job_configuration
task = job_farmer.Task(commands=[f'export MY_MESSAGE="Hello World!"',
'echo "${MY_MESSAGE}"'],
n_threads=1)
farmer.add_task(job_configuration.name, task)
And it builds the task list and the slurm configuration file:
farmer.build()
Finally, you can submit the jobs to the batch system using:
farmer.submit(dry_run = False)
(by default, dry_run = True
).
Note: By default, the job-farmer logs are not displayed.
To display them, you need to import the logging
package and set the log level to logging.INFO
such as:
logging.basicConfig(level=logging.INFO)
Support
If you are encountering an issue, please open a git issue: here
Contributing
Adding a new system or data center
Authors and acknowledgment
License
GNU Affero General Public License v3.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for job_farmer-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd9188b4e2cf7d4d74fe7843fe5b0b9519a29a9537d8726ef998c70c92bd7c4d |
|
MD5 | 87bc2da0f4b49c01852478e70b3eab35 |
|
BLAKE2b-256 | 822fc896a1d23955ef0057aae8915f6438915df8c01751ddbca3a2aa3d4ed751 |