Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, nodstream-plugin-k8s will be able to manage your kubernetes resources for you.
Installation
pip install nodestream-plugin-k8s
Usage
1. Create a nodestream.yaml file and project.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
2. Annotate your pipeline with nodestream-plugin-k8s annotations.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
3. Run nodestream-plugin-k8s to deploy your pipelines.
nodestream k8s sync --namespace my-nodestream-app-namespace
4. Check your kubernetes cluster for your deployed pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
5. Update your nodestream.yaml file and re-run nodestream-plugin-k8s to update your pipelines.
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
nodestream k8s sync --namespace my-nodestream-app-namespace
6. Check your kubernetes cluster for your updated pipelines.
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
7. Delete your pipelines.
nodestream k8s destroy --namespace my-nodestream-app-namespace
Annotations
nodestream_plugin_k8s_schedule
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
nodestream_plugin_k8s_concurrency
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
nodestream_plugin_k8s_image
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file nodestream_plugin_k8s-0.16.0.tar.gz.
File metadata
- Download URL: nodestream_plugin_k8s-0.16.0.tar.gz
- Upload date:
- Size: 6.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
06c3c89513df69aa2cb63a235dd95a6d5f7c24104395f0abce76791c7422e17e
|
|
| MD5 |
a02632374c14bb43e3cd7d5cc7a6c3c8
|
|
| BLAKE2b-256 |
da0f9c44928b9a64e216429c8a4fff2ffda8d2b285ccb3b3abba87a25b2d86d1
|
Provenance
The following attestation bundles were made for nodestream_plugin_k8s-0.16.0.tar.gz:
Publisher:
release.yaml on nodestream-proj/nodestream-plugin-k8s
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
nodestream_plugin_k8s-0.16.0.tar.gz -
Subject digest:
06c3c89513df69aa2cb63a235dd95a6d5f7c24104395f0abce76791c7422e17e - Sigstore transparency entry: 1944314953
- Sigstore integration time:
-
Permalink:
nodestream-proj/nodestream-plugin-k8s@3e02c6adf9a6314ed1d2058fd4fa5f0d178b9286 -
Branch / Tag:
refs/tags/0.16.0 - Owner: https://github.com/nodestream-proj
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yaml@3e02c6adf9a6314ed1d2058fd4fa5f0d178b9286 -
Trigger Event:
release
-
Statement type:
File details
Details for the file nodestream_plugin_k8s-0.16.0-py3-none-any.whl.
File metadata
- Download URL: nodestream_plugin_k8s-0.16.0-py3-none-any.whl
- Upload date:
- Size: 9.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
629c4598009cd34164958b79a773b630bae21d2b372269ad9c3f309a8a74d9b9
|
|
| MD5 |
a9881062f04a5232c6ebe853dc5f7a4f
|
|
| BLAKE2b-256 |
e2a830669bea56ee9091a4dba55b650bf9f72c9ea3d94c7816b3a47b218c3bb2
|
Provenance
The following attestation bundles were made for nodestream_plugin_k8s-0.16.0-py3-none-any.whl:
Publisher:
release.yaml on nodestream-proj/nodestream-plugin-k8s
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
nodestream_plugin_k8s-0.16.0-py3-none-any.whl -
Subject digest:
629c4598009cd34164958b79a773b630bae21d2b372269ad9c3f309a8a74d9b9 - Sigstore transparency entry: 1944315087
- Sigstore integration time:
-
Permalink:
nodestream-proj/nodestream-plugin-k8s@3e02c6adf9a6314ed1d2058fd4fa5f0d178b9286 -
Branch / Tag:
refs/tags/0.16.0 - Owner: https://github.com/nodestream-proj
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yaml@3e02c6adf9a6314ed1d2058fd4fa5f0d178b9286 -
Trigger Event:
release
-
Statement type:
