K8s Jobs is a library for implementing an asynchronous job server on Kubernetes. It is intended to provide a simple framework for executing single-shot asynchronous jobs or commands (unlike something like Celery that can have arbitrary fanout and nesting), as well as a server implementation that can stand in as a replacement for AWS Batch and trigger jobs on-command.
Kubernetes Job Management
This project provides an abstraction around kubernetes APIs to allow you to dynamically spawn (templated) jobs and clean up after them when they have run.
The two abstractions of interest are the JobManager and JobManagerFactory. The latter provides a factory for the former and helps convert (kubernetes) configuration into a working application.
The JobManager is responsible for creating (templated) jobs given a job definition name and template arguments. It is recommended that the jobs target a dedicated node instance group so as not to contend with live application resources. It is further recommended that you configure the Cluster Autoscaler on this instance group to ensure you do not run out of capacity (even better would be something like the Escalator, a batch job-oriented autoscaler). This will most closely mirror the behavior of a service like AWS Batch, which automatically adjusts the number of nodes based on workload.
The JobManager is also responsible for cleaning up terminated (completed or failed) jobs after some retention period. This is provided for version compatibility across K8s, and to avoid using a feature that is still in Alpha. For more details, see the TTL Controller.
Labeling and Annotations
The JobManager (and associated objects) makes use of labels and annotations in order to properly identify and manage jobs. Of note are the following:
Labels:
app.kubernetes.io/managed-by: A recommended kubernetes label, populated with the value of the JobSigner signature. This is used to logically identify jobs created by the JobManager of interest, rather than by third party applications or users.
job_definition_name: Identifies the job definition on which the job was based (maps to a name in the manager config).
Annotations:
job_deletion_time_unix_sec: If present, the earliest time at which the job can be deleted. It is only set after the job has reached a terminal state. This is meant to help implement baseline retention for resource management purposes, as well as to provide an avenue for users to mark and prevent the deletion of a job so that it can be inspected for debugging.
Examples
Flask Server
The server is a proof-of-concept implementation intended as a replacement for and extension to AWS Batch. It is a flask application housed completely under examples/flask. You do not need to use the server in order to take advantage of the primitives on which it relies.
The server listens on a route for job creation requests, much in the same way AWS batch might be implemented under the hood.
Kubernetes Resources
The Kubernetes resources under examples/k8s/ provide the configuration needed for deploying a server to Kubernetes. Specifically, it demonstrates how to configure jobs to be run by the manager. It relies on ConfigMap volume mounts in order to load the templates into a consistent location. See the JobManagerFactory for the specific required structure.
There is a corresponding dockerfile at examples/Dockerfile that can be used with the templates. You can build it as follows:
docker build -t flask-app -f examples/Dockerfile .QuickStart
To install dependencies:
poetry installTo run the sample server locally (make sure you have ~/.kube/config configured):
JOB_SIGNATURE=foo JOB_NAMESPACE=default JOB_DEFINITIONS_CONFIG_PATH=path/to/conf python examples/flask/app.py1 maintainer
Metadata
1 maintainer
Release files for k8s-jobs 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| k8s-jobs-0.2.0.tar.gz | 13.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| k8s_jobs-0.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 26.3 kB
Release files / k8s-jobs-0.2.0.tar.gz
| Download URL | k8s-jobs-0.2.0.tar.gz |
|---|---|
| Size | 13.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d3af00df2291b4ab55434224bcfba89548e8897d34b49da58994465f2325bc63
|
|
BLAKE2b-256 checksum How to use checksums |
0b0d639c501cc2cbfb43be905bd33616dede72a8b3abce93c94de90161ff68f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.2 CPython/3.8.2 Darwin/18.2.0
|
Release files / k8s_jobs-0.2.0-py3-none-any.whl
| Download URL | k8s_jobs-0.2.0-py3-none-any.whl |
|---|---|
| Size | 13.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
978fb454bbbd7997f0619c9af94efe711f001b438e40700475609422074257a4
|
|
BLAKE2b-256 checksum How to use checksums |
1c7d0d1923b05cbc2aa2f8600b47d7b5b549de033a8fdf95cb8acf1aec3481c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.2 CPython/3.8.2 Darwin/18.2.0
|
Security for k8s-jobs 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
