soopervisor 0.2.1

No project description provided

Soopervisor

Soopervisor introduces the concept of a Ploomber project, which is a standard way of running Ploomber pipelines.

Use cases

1. Running a pipeline locally

2. Running a pipeline in a continuous integration service

3. Scheduling a pipeline using cron (or Github Actions)

4. Running in Kubernetes via Argo workflows

5. Running in Apache Airflow

How it works

When running a pipeline, Soopervisor expects the following file layout:

1. environment.yml: Conda environment specification

2. pipeline.yaml: Ploomber pipeline specification

The parent folder to all these files is defined as the project’s root folder. The name of such folder is designed as the project’s name.

For example if your pipeline.yaml is located at /path/to/projects/some-project/pipeline.yaml, your project’s root folder is /path/to/projects/some-project and your project’s name is some-project.

If your project follows these two conventions, you’ll be able to use Soopervisor to run your project locally, continuous integration service or Apache Airflow.

Project validation

Before building/exporting your project, Soopervisor first checks that the project has the right structure, if it finds any issues, it reports them so you can fix them before you attempt to run the pipeline.

If all checks pass, it generates a bash script to install the conda environment and then run the pipeline.

How the script is used to actually execute the pipeline depends on your configuration settings, the simplest case is to just run it locally, but you can also tell Soopervisor to run the pipeline inside a Docker container or to just export your project to run in Kubernetes (using Argo) or Airflow.

Installation

pip install soopervisor

CHANGELOG

0.2.1 (2020-11-20)

• Adds Airflow DAG export

• Adds Argo/Kubernetes DAG export

• Support for uploading products to Box

0.2 (2020-10-15)

• Adds DockerExecutor

• Products are saved in a folder with the name of the current commit by default

• Conda environments are created locally in a .soopervisor/ folder

• Conda environments are cached by default

• Ability to customize arguments to ploomber build

0.1 (2020-08-09)

• First release