Skip to main content

A dagster plugin that allows you to run your Meltano project inside Dagster.

Project description

Dagster-meltano

A dagster plugin that allows you to run Meltano using Dagster.

Downloads

Installation

You can install using pip install dagster-meltano.

Examples

An example of automatically loading all jobs and schedules from your Meltano project.

from dagster import repository
from dagster_meltano import load_jobs_from_meltano_project

meltano_jobs = load_jobs_from_meltano_project("<path-to-meltano-root>")

@repository
def repository():
    return [meltano_jobs]

An example of running an abitrary meltano run command.

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]

You can inject Meltano config with the following Dagster config.

ops:
  tap_smoke_test_target_jsonl:
    config:
      env:
        TAP_SMOKE_TEST_STREAMS: '[{"stream_name": "new-stream", "input_filename": "demo.json"}]'

An example of running an arbitrary Meltano command.

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_command_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_command_job():
    meltano_command_op("install loader tap-smoke-test")()

@repository()
def repository():
    return [meltano_command_job]

Development using VSCode

  1. Open this repository in Visual Studio Code.
  2. Install the Remote - Containers plugin for Visual Studio Code.
  3. Go to the example Meltano project root cd meltano_project
  4. Install all plugins meltano install
  5. Start dagit meltano invoke dagster:start
  6. Visit localhost:3000 to access Dagit.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_meltano-1.5.5.tar.gz (7.7 kB view details)

Uploaded Source

Built Distribution

dagster_meltano-1.5.5-py3-none-any.whl (9.8 kB view details)

Uploaded Python 3

File details

Details for the file dagster_meltano-1.5.5.tar.gz.

File metadata

  • Download URL: dagster_meltano-1.5.5.tar.gz
  • Upload date:
  • Size: 7.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure

File hashes

Hashes for dagster_meltano-1.5.5.tar.gz
Algorithm Hash digest
SHA256 74120a425d3b96e1221923cd2ef9fe4716c933a4ea3518306530c466f4571bc3
MD5 0097adee0f47c7dfe8b9e35cf796c99b
BLAKE2b-256 2d71e99ff082598610198d20a8d24f2af59b26ea74adcb4c6c84ef1dcbe2d9f9

See more details on using hashes here.

File details

Details for the file dagster_meltano-1.5.5-py3-none-any.whl.

File metadata

  • Download URL: dagster_meltano-1.5.5-py3-none-any.whl
  • Upload date:
  • Size: 9.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure

File hashes

Hashes for dagster_meltano-1.5.5-py3-none-any.whl
Algorithm Hash digest
SHA256 55152ffab555850bc2046bb792cbe6176d92675ead637be7a1f204368a078d60
MD5 30f8f500efcf22a90bf9097d43a5de05
BLAKE2b-256 fec408e07f80773329c7429b5c28d55f0d97c273b4875fe10eb9c6280541860f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page