Skip to main content

Async Job Manager + AWS Batch Replacement for K8s

Project description

https://travis-ci.com/danieltahara/k8s-jobs.svg?token=cZTmQ2jMoLFe6Ve33X6M&branch=master

K8s Jobs is a library for implementing an asynchronous job server on Kubernetes. It is intended to provide a simple framework for executing single-shot asynchronous jobs or commands (unlike something like Celery that can have arbitrary fanout and nesting), as well as a server implementation that can stand in as a replacement for AWS Batch and trigger jobs on-command.

What’s Inside

Kubernetes Job Management

This project provides an abstraction around kubernetes APIs to allow you to dynamically spawn (templated) jobs and clean up after them when they have run.

The two abstractions of interest are the JobManager and JobManagerFactory. The latter provides a factory for the former and helps convert (kubernetes) configuration into a working application.

The JobManager is responsible for creating (templated) jobs given a job definition name and template arguments. It is recommended that the jobs target a dedicated node instance group so as not to contend with live application resources. It is further recommended that you configure the Cluster Autoscaler on this instance group to ensure you do not run out of capacity (even better would be something like the Escalator, a batch job-oriented autoscaler). This will most closely mirror the behavior of a service like AWS Batch, which automatically adjusts the number of nodes based on workload.

The JobManager is also responsible for cleaning up terminated (completed or failed) jobs after some retention period. This is provided for version compatibility across K8s, and to avoid using a feature that is still in Alpha. For more details, see the TTL Controller.

Example Flask Server

The server is a proof-of-concept impleemtnation intended as a replacement for and extension to AWS Batch. It is a flask application housed completely under examples/flask. You do not need to use the server in order to take advantage of the primitives on which it relies.

The server listens on a route for job creation requests, much in the same way AWS batch might be implemented under the hood.

Kubernetes Resources

The Kubernetes resources under examples/k8s/ provide the configuration needed for deploying a server to Kubernetes. Specifically, it demonstrates how to configure jobs to be run by the manager. It relies on ConfigMap volume mounts in order to load the templates into a consistent location. See the JobManagerFactory for the specific required structure.

There is a corresponding dockerfile at examples/Dockerfile that can be used with the templates. You can build it as follows:

docker build -t flask-app -f examples/Dockerfile .

QuickStart

To install dependencies:

poetry install

To run the sample server locally (make sure you have ~/.kube/config configured):

JOB_SIGNATURE=foo JOB_NAMESPACE=default JOB_DEFINITIONS_CONFIG_PATH=path/to/conf python examples/flask/app.py

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

k8s-jobs-0.1.5.tar.gz (10.8 kB view details)

Uploaded Source

Built Distribution

k8s_jobs-0.1.5-py3-none-any.whl (10.8 kB view details)

Uploaded Python 3

File details

Details for the file k8s-jobs-0.1.5.tar.gz.

File metadata

  • Download URL: k8s-jobs-0.1.5.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/0.12.16 CPython/3.6.8 Linux/4.15.0-1051-aws

File hashes

Hashes for k8s-jobs-0.1.5.tar.gz
Algorithm Hash digest
SHA256 ce518c14783b87fe5eaedec2fd793401880f624aa77bec648b33aa9dcf0097ce
MD5 5b286453bb36de7b73c14b90922a27ae
BLAKE2b-256 a08ed2c98edcf3324dd86d1a361aa0530570c93d4c90fbf71c5e25eced46155c

See more details on using hashes here.

File details

Details for the file k8s_jobs-0.1.5-py3-none-any.whl.

File metadata

  • Download URL: k8s_jobs-0.1.5-py3-none-any.whl
  • Upload date:
  • Size: 10.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/0.12.16 CPython/3.6.8 Linux/4.15.0-1051-aws

File hashes

Hashes for k8s_jobs-0.1.5-py3-none-any.whl
Algorithm Hash digest
SHA256 b6ae53033080ada11a29a222771a5761c88429645d27020a0b7d65ce3d2535b9
MD5 67c4057990196cdfa88ca6e9a31c568a
BLAKE2b-256 6e3b58eb2de683e3089960c793b63be8219c3d673c0418b761ae95e19cbc108c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page