Kedro-Argo
Converting kedro pipelines to argo pipelines.
Free software: BSD 3-Clause License
Installation
pip install kedro-argo
You can also install the in-development version with:
pip install https://github.com/nraw/kedro-argo/archive/master.zip
Requirements
To be used with Kedro, so it’s assumed this package is used with a Kedro project.
Argo CLI is needed for the deployment step. It’s also assumed that Argo is already installed on your kuberentes instance.
You must specify an image name as a parameter. You can generate the image using Kedro-docker.
As the workflow will be in Argo, which means every step will run in its own container. Therefore, all datasets should be somehow passed between containers or else the pipeline will fail. This means either all datasets should be saved externally (S3, Azure, etc.) or in a shared folder that your deployment would have access to.
Usage
When installed, argo should be visible under your kedro commands, if you’re in a kedro project
kedro
Then you have two options for obtaining the yaml file, namely via Helm or via ytt.
Helm
kedro argo IMAGE_NAME
Add this repository to your helm charts:
helm repo add kedro-argo https://nraw.github.io/kedro-argo-helm/
Then either directly install it by passing the kedro.yaml for input values
helm install -f templates/kedro.yaml kedro-argo kedro-argo/kedro-argo
Or clone it to your repository and change anything that you would still need:
helm pull kedro-argo/kedro-argo --untar
ytt
Get the kedro.yaml file by running
kedro argo --ytt IMAGE_NAME
You can now run:
ytt -f templates > argo.yaml
or if you prefer in Docker:
docker run --rm -it --name ytt -v $(pwd)/templates:/templates gerritk/ytt:latest -f /templates > argo.yaml
and finally
argo submit --watch argo.yaml
Documentation
Development
To run the all tests run:
tox
Note, to combine the coverage data from all the tox environments run:
Windows |
set PYTEST_ADDOPTS=--cov-append tox |
|---|---|
Other |
PYTEST_ADDOPTS=--cov-append tox |
Changelog
0.0.8 (2020-04-27)
Changed the ytt option to be a flag
0.0.7 (2020-03-27)
Changed the default templating option to be Helm instead of ytt
0.0.5 (2020-03-08)
Dirty names are now transformed to make less likely that symbols break Argo
0.0.4 (2020-03-07)
Refactoring and initial adaptation
Inclusion of tests
0.0.0 (2020-03-07)
First release on PyPI.
1 maintainer
Metadata
1 maintainer
Release files for kedro-argo 0.0.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| kedro-argo-0.0.9.tar.gz | 264.0 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kedro_argo-0.0.9-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 270.8 kB
Release files / kedro-argo-0.0.9.tar.gz
| Download URL | kedro-argo-0.0.9.tar.gz |
|---|---|
| Size | 264.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7be51703b42efdedb13cc71731a14d5f1f7ca8b091f184a209e02992da740949
|
|
BLAKE2b-256 checksum How to use checksums |
a339fa70a3e22ba59c80d2788b5230456bf390cc0ce959fa6e9852ea22e10f74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.3.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9
|
Release files / kedro_argo-0.0.9-py2.py3-none-any.whl
| Download URL | kedro_argo-0.0.9-py2.py3-none-any.whl |
|---|---|
| Size | 6.8 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
41620624021d654f7d6206411c1663ad737dd68fde601690a5ffd660fc09201a
|
|
BLAKE2b-256 checksum How to use checksums |
e31c7619c845fddf9ac1fc1290731ec71fd587ee681b5b199ceeef5cd961854e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.3.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9
|
Security for kedro-argo 0.0.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
