A plugin for nodestream that manages k8s resources.
Project description
Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, nodstream-plugin-k8s
will be able to manage your kubernetes resources for you.
Installation
pip install nodestream-plugin-k8s
Usage
1. Create a nodestream.yaml
file and project.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
2. Annotate your pipeline with nodestream-plugin-k8s
annotations.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
3. Run nodestream-plugin-k8s
to deploy your pipelines.
nodestream k8s sync --namespace my-nodestream-app-namespace
4. Check your kubernetes cluster for your deployed pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
5. Update your nodestream.yaml
file and re-run nodestream-plugin-k8s
to update your pipelines.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
nodestream k8s sync --namespace my-nodestream-app-namespace
6. Check your kubernetes cluster for your updated pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
7. Delete your pipelines.
nodestream k8s destroy --namespace my-nodestream-app-namespace
Annotations
nodestream_plugin_k8s_schedule
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
nodestream_plugin_k8s_concurrency
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
nodestream_plugin_k8s_image
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nodestream_plugin_k8s-0.13.0b1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebeea45e7cc335b9152b409e86b923e6c49ef692358795703b5b9cd0db9b736f |
|
MD5 | 50da7705c5552d88754c3c8fd03010ef |
|
BLAKE2b-256 | c618860a76be2401a30d578e472cb382be3411ff40309cca08c08b422ffa932f |
Hashes for nodestream_plugin_k8s-0.13.0b1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9347d5f38860bd6c721d8debd92d3bce7631511197b0628f28b6eca1f56f3bfb |
|
MD5 | ecdceb4ac7f49f2e8d6a8b7a7559564a |
|
BLAKE2b-256 | a0bd0d4f8436ef9d406c1bd6961531d54c8e484fcf28204254e6bc0c8b0bc89f |