Library to create Dagster jobs from YAML
Project description
Dagster Composable Graphs
Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.
This library provides additional functionality to define dagster jobs from a
file in .yaml
format and may be used with any other package that integrates
with dagster.
Visit the documentation here.
Partially inspired by post Abstracting Pipelines for Analysts with a YAML DSL on the dagster blog.
Example
Consider the following definition of a ComposableGraph
. Notice in particular
sections inputs
, operations
and dependencies
. Respectively these define
the graph inputs, which dagster
ops or
graphs are
part of the job, and their dependencies.
apiVersion: truevoid.dev/v1alpha1
kind: ComposableGraph
metadata:
name: concatenate-graphs
spec:
inputs:
x: 2.0
y: 5.0
operations:
- name: add_and_multiply
function: example.jobs.add_and_multiply
- name: add
function: example.jobs.add
- name: multiply
function: example.jobs.multiply
dependencies:
- name: add_and_multiply
inputs:
- x
- y
- name: add
inputs:
- node: add_and_multiply
pointer: /add
- x
- name: multiply
inputs:
- y
- node: add_and_multiply
pointer: /multiply
Results in the following job, visualized using dagster webserver UI:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dagster_composable_graphs-0.1.2.tar.gz
.
File metadata
- Download URL: dagster_composable_graphs-0.1.2.tar.gz
- Upload date:
- Size: 9.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bfed7dbf99c240541f88027d2e38f208b10456034aac0573ad3b7a80f9bb390 |
|
MD5 | 93e5eb1941b0dd6eb536f37186840c22 |
|
BLAKE2b-256 | 091168388b32660544268dc2c6122f15d321d961d2bbe737a80a78e1a1183fd4 |
File details
Details for the file dagster_composable_graphs-0.1.2-py3-none-any.whl
.
File metadata
- Download URL: dagster_composable_graphs-0.1.2-py3-none-any.whl
- Upload date:
- Size: 11.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | afb58994bc49a0cf40d411f5857549fe783cf6b3d7775f92c25c0a44a465e36b |
|
MD5 | 5ed23b1c44d439b2b47aa12f909a4118 |
|
BLAKE2b-256 | 7e545d46a2daeb0418939a02aa60d160df0872631d8fa964641e7ad8cb4c35ee |