A plugin for nodestream that manages k8s resources.
Project description
Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, nodstream-plugin-k8s
will be able to manage your kubernetes resources for you.
Installation
pip install nodestream-plugin-k8s
Usage
1. Create a nodestream.yaml
file and project.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
2. Annotate your pipeline with nodestream-plugin-k8s
annotations.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
3. Run nodestream-plugin-k8s
to deploy your pipelines.
nodestream k8s sync --namespace my-nodestream-app-namespace
4. Check your kubernetes cluster for your deployed pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
5. Update your nodestream.yaml
file and re-run nodestream-plugin-k8s
to update your pipelines.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
nodestream k8s sync --namespace my-nodestream-app-namespace
6. Check your kubernetes cluster for your updated pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
7. Delete your pipelines.
nodestream k8s destroy --namespace my-nodestream-app-namespace
Annotations
nodestream_plugin_k8s_schedule
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
nodestream_plugin_k8s_concurrency
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
nodestream_plugin_k8s_image
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nodestream_plugin_k8s-0.12.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e6f95fdbf8004852b3142a618e9b48090d0423069c3c636314b1bf12179a7b15 |
|
MD5 | a96c404c8291ae534f94adf8fd35cb63 |
|
BLAKE2b-256 | 2c1b4d116dfabcc72588f572cc95f8a5e90f901dfc7709b2e30f74a19e751e2d |
Hashes for nodestream_plugin_k8s-0.12.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd8551e3ccc8f85d8ed0f1964a7ec6b14cab699753f129f1e24bc3f0000156c8 |
|
MD5 | 8d870315347e755b7214838795db795e |
|
BLAKE2b-256 | 1b89814c6ff7e64b1d2da5dced31f7fff0a5d194993b08b65239057f71f329e3 |