Skip to main content

Converting kedro pipelines to argo pipelines.

Project description

Converting kedro pipelines to argo pipelines.

docs/images/kedro-argo.png
  • Free software: BSD 3-Clause License

Installation

pip install kedro-argo

You can also install the in-development version with:

pip install https://github.com/nraw/kedro-argo/archive/master.zip

Requirements

  • To be used with Kedro, so it’s assumed this package is used with a Kedro project.

  • Argo CLI is needed for the deployment step. It’s also assumed that Argo is already installed on your kuberentes instance.

  • You must specify an image name as a parameter. You can generate the image using Kedro-docker.

  • As the workflow will be in Argo, which means every step will run in its own container. Therefore, all datasets should be somehow passed between containers or else the pipeline will fail. This means either all datasets should be saved externally (S3, Azure, etc.) or in a shared folder that your deployment would have access to.

Usage

When installed, argo should be visible under your kedro commands, if you’re in a kedro project

kedro

Then you have two options for obtaining the yaml file, namely via Helm or via ytt.

Helm

kedro argo IMAGE_NAME

Add this repository to your helm charts:

helm repo add kedro-argo https://nraw.github.io/kedro-argo-helm/

Then either directly install it by passing the kedro.yaml for input values

helm install -f templates/kedro.yaml kedro-argo kedro-argo/kedro-argo

Or clone it to your repository and change anything that you would still need:

helm pull kedro-argo/kedro-argo --untar

ytt

Get the kedro.yaml file by running

kedro argo --ytt IMAGE_NAME

You can now run:

ytt -f templates > argo.yaml

or if you prefer in Docker:

docker run --rm -it --name ytt -v $(pwd)/templates:/templates gerritk/ytt:latest -f /templates > argo.yaml

and finally

argo submit --watch argo.yaml

Documentation

https://kedro-argo.readthedocs.io/

Development

To run the all tests run:

tox

Note, to combine the coverage data from all the tox environments run:

Windows

set PYTEST_ADDOPTS=--cov-append
tox

Other

PYTEST_ADDOPTS=--cov-append tox

Changelog

0.0.8 (2020-04-27)

  • Changed the ytt option to be a flag

0.0.7 (2020-03-27)

  • Changed the default templating option to be Helm instead of ytt

0.0.5 (2020-03-08)

  • Dirty names are now transformed to make less likely that symbols break Argo

0.0.4 (2020-03-07)

  • Refactoring and initial adaptation

  • Inclusion of tests

0.0.0 (2020-03-07)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kedro-argo-0.0.9.tar.gz (264.0 kB view details)

Uploaded Source

Built Distribution

kedro_argo-0.0.9-py2.py3-none-any.whl (6.8 kB view details)

Uploaded Python 2Python 3

File details

Details for the file kedro-argo-0.0.9.tar.gz.

File metadata

  • Download URL: kedro-argo-0.0.9.tar.gz
  • Upload date:
  • Size: 264.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.3.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for kedro-argo-0.0.9.tar.gz
Algorithm Hash digest
SHA256 7be51703b42efdedb13cc71731a14d5f1f7ca8b091f184a209e02992da740949
MD5 c28cc894552e44f305b0b3441b76b083
BLAKE2b-256 a339fa70a3e22ba59c80d2788b5230456bf390cc0ce959fa6e9852ea22e10f74

See more details on using hashes here.

File details

Details for the file kedro_argo-0.0.9-py2.py3-none-any.whl.

File metadata

  • Download URL: kedro_argo-0.0.9-py2.py3-none-any.whl
  • Upload date:
  • Size: 6.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/45.3.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for kedro_argo-0.0.9-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 41620624021d654f7d6206411c1663ad737dd68fde601690a5ffd660fc09201a
MD5 ba17ba1e180284d3577d0347978d712f
BLAKE2b-256 e31c7619c845fddf9ac1fc1290731ec71fd587ee681b5b199ceeef5cd961854e

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page