Skip to main content

No project description provided

Project description

Soopervisor

CI badge CI macOS badge CI Windows badge

Soopervisor runs Ploomber pipelines for batch processing (large-scale training or batch serving) or online inference.

pip install soopervisor

Check out the documentation to learn more.

Compatible with Python 3.6 and higher.

Supported platforms

  • Batch serving and large-scale training:

    • Kubernetes / Argo Workflows

    • AWS Batch

  • Online inference:

    • AWS Lambda

Usage

Say that you want to train multiple models in a Kubernetes cluster, you may create a new target environment to execute your pipeline using Argo Workflows:

soopervisor add training --backend argo-workflows

After filling in some basic configuration settings, export the pipeline with:

soopervisor export training

Depending on the selected backend (Argo, Airflow, AWS Batch or AWS Lambda), configuration details will change but the API remains the same: soopervisor add, then soopervisor export. CHANGELOG =========

0.5.2 (2022-01-02)

  • Clearer error message when pending git commit

  • Clearer error message when the user does not change docker repository default value (#29)

  • Argo spec sets imagePullPolicy to Never if repository is null

  • Documents Kubernetes/Argo configuration schema

  • General documentation improvements

0.5.1 (2021-07-26)

  • Better error message when lock files do not exist

  • Documentation note on when using shared disks (must pass --skip-tests)

  • Adds build as a dependency

  • Check for lock files before creating soopervisor.yaml

  • Fixes docker.build issue that caused a pipeline.yaml to be used even when the environment required one with another name

  • Fixes error that caused AWS batch args to be passed as a single str

0.5 (2021-07-09)

  • load_tasks tries to initialize a spec matching the target env name (e.g., training target looks for pipeline.train.yaml

  • Compatibility fixes with Ploomber (requires >=0.12.1)

  • Fixes an error that caused Dockerfile to include a line to install project as a package even if there was no setup.py file

0.4.2 (2021-06-04)

  • Adds exclude to ignore files/directories from docker image

  • Adds user guide section to documentation

0.4.1 (2021-05-31)

  • Adds --mode option to soopervisor export

  • Batch export stops if there are no tasks to execute

  • Adds --skip-tests option to skip tests before submitting

0.4 (2021-05-22)

Important: Soopervisor was re-written. Some modules were deprecated and the API changed. This new architecture allows us to greatly simplify user experience and easily incorporate more platforms in the future.

  • New CLI

  • New documentation

  • New (simplified) soopervisor.yaml configuration schema

  • Support for non-packaged projects (i.e., the ones without a setup.py file)

  • Support for AWS Batch

  • Support for AWS Lambda

  • Argo Workflows integration builds a docker image

  • Airflow integration produces a DAG with DockerOperator tasks

  • Deprecates build module

  • Deprecates script module

  • Deprecates Box integration

0.3.4 (2021-04-18)

  • Export projects compatible with ploomber.OnlineModel to AWS Lambda

  • Allow initialization from empty soopervisor.yaml

0.3.3 (2021-03-07)

  • Support to pass extra cli args to ploomber task (via args in soopervisor.yaml) when running in Argo and Airflow

0.3.2 (2021-02-13)

  • Adds --root arg to soopervisor export-airflow to select an alternative project’s root

  • Determines default entry point using Ploomber’s API to allow automated discovery of pipeline.yaml in package layouts (e.g. src/package/pipeline.yaml)

0.3.1 (2021-02-11)

  • Changes to the Airflow generated DAG

  • Fixes a bug when initializing configuration from projects whose root is not the current directory

0.3 (2021-01-24)

  • env.airflow.yaml optional when exporting to Airflow (#17)

  • Validating exported argo YAML spec

  • Output argo YAML spec displays script in literal mode to make it readable

  • Fixed extra whitespace in generated script

  • Refactors ArgoMountedVolume to provide flexibility for different types of k8s volumes

  • Adds section in the documentation to run examples using minikube

  • Adds a few echo statements to generated script to provide better status feedback

0.2.2 (2020-11-21)

  • Adds ability to skip dag loading during project validation

  • Box uploader imported only if needed

  • Exposes option to skip dag loading from the CLI

0.2.1 (2020-11-20)

  • Adds Airflow DAG export

  • Adds Argo/Kubernetes DAG export

  • Support for uploading products to Box

0.2 (2020-10-15)

  • Adds DockerExecutor

  • Products are saved in a folder with the name of the current commit by default

  • Conda environments are created locally in a .soopervisor/ folder

  • Conda environments are cached by default

  • Ability to customize arguments to ploomber build

0.1 (2020-08-09)

  • First release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

soopervisor-0.5.2.tar.gz (28.1 kB view hashes)

Uploaded Source

Built Distribution

soopervisor-0.5.2-py3-none-any.whl (35.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page