A plugin for nodestream that manages k8s resources.
Project description
Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, nodstream-plugin-k8s
will be able to manage your kubernetes resources for you.
Installation
pip install nodestream-plugin-k8s
Usage
1. Create a nodestream.yaml
file and project.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
2. Annotate your pipeline with nodestream-plugin-k8s
annotations.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
3. Run nodestream-plugin-k8s
to deploy your pipelines.
nodestream k8s sync --namespace my-nodestream-app-namespace
4. Check your kubernetes cluster for your deployed pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
5. Update your nodestream.yaml
file and re-run nodestream-plugin-k8s
to update your pipelines.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
nodestream k8s sync --namespace my-nodestream-app-namespace
6. Check your kubernetes cluster for your updated pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
7. Delete your pipelines.
nodestream k8s destroy --namespace my-nodestream-app-namespace
Annotations
nodestream_plugin_k8s_schedule
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
nodestream_plugin_k8s_concurrency
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
nodestream_plugin_k8s_image
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nodestream_plugin_k8s-0.13.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5042bcfbca812f03ff99fa37319b7978cd6fce9097e7fc75a2415bdcbd43a1ea |
|
MD5 | 7c93a803e51ccae286334768dd1b9c0d |
|
BLAKE2b-256 | fc34d478e4c22019259b6b8b75da8a1b85f109f5acbdfd0e3e4a800d95bfd71f |
Hashes for nodestream_plugin_k8s-0.13.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34946e04c184a1a5d381925a821d66275753414913013923823354834375e428 |
|
MD5 | 42c9ce0ccf2220820fd1c19a2e8adcf4 |
|
BLAKE2b-256 | 6244bb8b2be289e459ae08a00741faf373f8c7624ba5552852c12ef4e25f0299 |