Dagster-meltano
A dagster plugin that allows you to run Meltano using Dagster.
Installation
You can install using pip install dagster-meltano.
Examples
An example of automatically loading all jobs and schedules from your Meltano project.
from dagster import repository
from dagster_meltano import load_jobs_from_meltano_project
meltano_jobs = load_jobs_from_meltano_project("<path-to-meltano-root>")
@repository
def repository():
return [meltano_jobs]
An example of running an abitrary meltano run command.
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op
@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
tap_done = meltano_run_op("tap-1 target-1")()
meltano_run_op("tap-2 target-2")(tap_done)
@repository()
def repository():
return [meltano_run_job]
You can inject Meltano config with the following Dagster config.
ops:
tap_smoke_test_target_jsonl:
config:
env:
TAP_SMOKE_TEST_STREAMS: '[{"stream_name": "new-stream", "input_filename": "demo.json"}]'
An example of running an arbitrary Meltano command.
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_command_op
@job(resource_defs={"meltano": meltano_resource})
def meltano_command_job():
meltano_command_op("install loader tap-smoke-test")()
@repository()
def repository():
return [meltano_command_job]
Development using VSCode
- Open this repository in Visual Studio Code.
- Install the Remote - Containers plugin for Visual Studio Code.
- Go to the example Meltano project root
cd meltano_project - Install all plugins
meltano install - Start dagit
meltano invoke dagster:start - Visit
localhost:3000to access Dagit.
1 maintainer
Metadata
1 maintainer
Release files for dagster-meltano 1.5.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dagster_meltano-1.5.5.tar.gz | 7.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dagster_meltano-1.5.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 17.5 kB
Release files / dagster_meltano-1.5.5.tar.gz
| Download URL | dagster_meltano-1.5.5.tar.gz |
|---|---|
| Size | 7.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
74120a425d3b96e1221923cd2ef9fe4716c933a4ea3518306530c466f4571bc3
|
|
BLAKE2b-256 checksum How to use checksums |
2d71e99ff082598610198d20a8d24f2af59b26ea74adcb4c6c84ef1dcbe2d9f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure
|
Release files / dagster_meltano-1.5.5-py3-none-any.whl
| Download URL | dagster_meltano-1.5.5-py3-none-any.whl |
|---|---|
| Size | 9.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
55152ffab555850bc2046bb792cbe6176d92675ead637be7a1f204368a078d60
|
|
BLAKE2b-256 checksum How to use checksums |
fec408e07f80773329c7429b5c28d55f0d97c273b4875fe10eb9c6280541860f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.4 CPython/3.10.12 Linux/6.5.0-1025-azure
|
