Library to create Dagster jobs from YAML
Project description
Dagster Composable Graphs
Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.
This library provides additional functionality to define dagster jobs from a
file in .yaml
format and may be used with any other package that integrates
with dagster.
Visit the documentation here.
Partially inspired by post Abstracting Pipelines for Analysts with a YAML DSL on the dagster blog.
Example
Consider the following definition of a ComposableGraph
. Notice in particular
sections inputs
, operations
and dependencies
. Respectively these define
the graph inputs, which dagster
ops or
graphs are
part of the job, and their dependencies.
apiVersion: truevoid.dev/v1alpha1
kind: ComposableGraph
metadata:
name: concatenate-graphs
spec:
inputs:
x: 2.0
y: 5.0
operations:
- name: add_and_multiply
function: example.jobs.add_and_multiply
- name: add
function: example.jobs.add
- name: multiply
function: example.jobs.multiply
dependencies:
- name: add_and_multiply
inputs:
- x
- y
- name: add
inputs:
- node: add_and_multiply
pointer: /add
- x
- name: multiply
inputs:
- y
- node: add_and_multiply
pointer: /multiply
Results in the following job, visualized using dagster webserver UI:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dagster_composable_graphs-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bfed7dbf99c240541f88027d2e38f208b10456034aac0573ad3b7a80f9bb390 |
|
MD5 | 93e5eb1941b0dd6eb536f37186840c22 |
|
BLAKE2b-256 | 091168388b32660544268dc2c6122f15d321d961d2bbe737a80a78e1a1183fd4 |
Hashes for dagster_composable_graphs-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afb58994bc49a0cf40d411f5857549fe783cf6b3d7775f92c25c0a44a465e36b |
|
MD5 | 5ed23b1c44d439b2b47aa12f909a4118 |
|
BLAKE2b-256 | 7e545d46a2daeb0418939a02aa60d160df0872631d8fa964641e7ad8cb4c35ee |