Kubernetes-job: simple Kubernetes job creation

A library for starting a Kubernetes batch job as a normal Python function call.

For source code and tickets, see our project page on Gitlab. The documentation is hosted on ReadTheDocs.
Kubernetes-job can be found on Pypi for easy installation with pip.

Installation

Installation with Pip:

pip install kubernetes-job

Quick start

from kubernetes_job import JobManager


def add(a, b):
    return a + b


manager = JobManager(k8s_client=k8s_client, k8s_job_spec='job.yaml', namespace='default')
job = manager.create_job(add, 1, 2)

The JobManager will now create a Kubernetes job using the basic job specification in the job.yaml file. The call to add is then passed on to the new job node, where the function is subsequently executed.

The job.yaml file should be adjusted to your needs. This is the place to put Kubernetes node selectors, Docker base images, etc. etc. Please refer to the Kubernetes documentation for details.

Please note: this is a very silly example, for two obvious reasons.

First, add will take a very short time to complete, and is therefore not a function you would want to spawn a Kubernetes job for. A job should be created for a task that is not easily performed on the calling machine. A good example would be training Machine Learning models on a heavy CUDA node, started from a web server node with modest resources.

Second, Kubernetes jobs do not return values! This means the result of this addition will be lost. In a Kubernetes job, it is up to the job to save its work. In this case, the result of (1 + 2) will be lost for humanity.

Please see the examples and the test/ directory.

API usage

Initializing the JobManager

The JobManager must be supplied a yaml template file (see above) and the Kubernetes client.

from pathlib import Path
from kubernetes_job import JobManager

# Path to worker configuration
yaml_spec = Path(__file__).parent / 'job.yml'

# initialize the job manager
manager = JobManager(k8s_client=k8s_client, k8s_job_spec=yaml_spec, namespace='default')

The k8s_job_spec may be a path to a file, or a dict instance. The latter is handy for generating configuration on the fly!

JobManager also needs a Kubernetes client. More information about how to connect to Kubernetes can be found here.

Creating a new job

A job can be started by invoking create_job on the JobManager instance:

# function to pass to the job
def add(a, b):
    result = a + b
    print(result)
    return result

# create a new job
job = manager.create_job(add, 123, 456)

create_job takes a function pointer. This function pointer and all arguments (*args and **kwargs) are then "pickled", and merged in the job template.

Our job is now running on the Kubernetes cluster!

Listing jobs

# list all jobs
for job in manager.list_jobs():
    print(f"Found: {job.metadata.name}")

Retrieving job status

from kubernetes_job import is_active, is_succeeded, is_failed, is_completed, job_status 

# get the status of a job
job = manager.read_job(name)

print(f"Status: {job_status(job)}")
print(f"Running: {is_active(job)} Completed: {is_completed(job)}")
print(f"Succeeded: {is_succeeded(job)} Failed: {is_failed(job)}")

Cleaning up finished jobs

# cleaning up finished jobs
manager.cleanup_jobs()

Deleting jobs

# delete a job
manager.delete_job(name)