A plugin for nodestream that manages k8s resources.
Project description
Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, nodstream-plugin-k8s
will be able to manage your kubernetes resources for you.
Installation
pip install nodestream-plugin-k8s
Usage
1. Create a nodestream.yaml
file and project.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
2. Annotate your pipeline with nodestream-plugin-k8s
annotations.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
3. Run nodestream-plugin-k8s
to deploy your pipelines.
nodestream k8s sync --namespace my-nodestream-app-namespace
4. Check your kubernetes cluster for your deployed pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
5. Update your nodestream.yaml
file and re-run nodestream-plugin-k8s
to update your pipelines.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
nodestream k8s sync --namespace my-nodestream-app-namespace
6. Check your kubernetes cluster for your updated pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
7. Delete your pipelines.
nodestream k8s destroy --namespace my-nodestream-app-namespace
Annotations
nodestream_plugin_k8s_schedule
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
nodestream_plugin_k8s_concurrency
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
nodestream_plugin_k8s_image
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nodestream_plugin_k8s-0.8.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 631196e9b63f753a0f72cd7fd10c8f769a4ab48cabef7f2179abb5ff00af7ab3 |
|
MD5 | a5e543f02da07ac74069524eea1ab292 |
|
BLAKE2b-256 | 93a7d1a6e90c5d9f7a9382147016e3db39f0c00e520de1ad4c38880d6ea83e6a |
Hashes for nodestream_plugin_k8s-0.8.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20d08171a48170d554a06126c519817860816ae991d2e6f7699fff683694b568 |
|
MD5 | 9fd65e5daa57f7d56a09aae1cdfac49c |
|
BLAKE2b-256 | 432a5789cabff8e0b0c93f1a50be551605d8d3a194e7b31840417ebdc11a481b |