Skip to main content

A dagster plugin that allows you to run your Meltano project inside Dagster.

Project description

Dagster-meltano

A dagster plugin that allows you to run Meltano using Dagster.

Downloads

Installation

You can install using pip install dagster-meltano.

Examples

An example of automatically loading all jobs and schedules from your Meltano project.

from dagster import repository
from dagster_meltano import load_jobs_from_meltano_project

meltano_jobs = load_jobs_from_meltano_project("<path-to-meltano-root>")

@repository
def repository():
    return [meltano_jobs]

An example of running an abitrary meltano run command.

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]

You can inject Meltano config with the following Dagster config.

ops:
  tap_smoke_test_target_jsonl:
    config:
      env:
        TAP_SMOKE_TEST_STREAMS: '[{"stream_name": "new-stream", "input_filename": "demo.json"}]'

An example of running an arbitrary Meltano command.

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_command_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_command_job():
    meltano_command_op("install loader tap-smoke-test")()

@repository()
def repository():
    return [meltano_command_job]

Development using VSCode

  1. Open this repository in Visual Studio Code.
  2. Install the Remote - Containers plugin for Visual Studio Code.
  3. Go to the example Meltano project root cd meltano_project
  4. Install all plugins meltano install
  5. Start dagit meltano invoke dagster:start
  6. Visit localhost:3000 to access Dagit.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_meltano-1.5.2.tar.gz (7.7 kB view hashes)

Uploaded Source

Built Distribution

dagster_meltano-1.5.2-py3-none-any.whl (9.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page