Skip to main content

Library to create Dagster jobs from YAML

Project description

Dagster Composable Graphs

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

This library provides additional functionality to define dagster jobs from a file in .yaml format and may be used with any other package that integrates with dagster.

Visit the documentation here.

Partially inspired by post Abstracting Pipelines for Analysts with a YAML DSL on the dagster blog.

Example

Consider the following definition of a ComposableGraph. Notice in particular sections inputs, operations and dependencies. Respectively these define the graph inputs, which dagster ops or graphs are part of the job, and their dependencies.

apiVersion: truevoid.dev/v1alpha1
kind: ComposableGraph
metadata:
  name: concatenate-graphs
spec:
  inputs:
    x: 2.0
    y: 5.0
  operations:
    - name: add_and_multiply
      function: example.jobs.add_and_multiply
    - name: add
      function: example.jobs.add
    - name: multiply
      function: example.jobs.multiply
  dependencies:
    - name: add_and_multiply
      inputs:
        - x
        - y
    - name: add
      inputs:
        - node: add_and_multiply
          pointer: /add
        - x
    - name: multiply
      inputs:
        - y
        - node: add_and_multiply
          pointer: /multiply

Results in the following job, visualized using dagster webserver UI:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_composable_graphs-0.1.2.tar.gz (9.9 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file dagster_composable_graphs-0.1.2.tar.gz.

File metadata

File hashes

Hashes for dagster_composable_graphs-0.1.2.tar.gz
Algorithm Hash digest
SHA256 3bfed7dbf99c240541f88027d2e38f208b10456034aac0573ad3b7a80f9bb390
MD5 93e5eb1941b0dd6eb536f37186840c22
BLAKE2b-256 091168388b32660544268dc2c6122f15d321d961d2bbe737a80a78e1a1183fd4

See more details on using hashes here.

File details

Details for the file dagster_composable_graphs-0.1.2-py3-none-any.whl.

File metadata

File hashes

Hashes for dagster_composable_graphs-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 afb58994bc49a0cf40d411f5857549fe783cf6b3d7775f92c25c0a44a465e36b
MD5 5ed23b1c44d439b2b47aa12f909a4118
BLAKE2b-256 7e545d46a2daeb0418939a02aa60d160df0872631d8fa964641e7ad8cb4c35ee

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page