dagster-malloy

dagster-malloy is an unofficial community integration library providing Dagster support for Malloy models (.malloy) and notebooks (.malloynb).

Features

• Malloy as Dagster assets: Expose Malloy queries, dashboards and notebooks as Dagster assets including rich metadata (compiled SQL, Malloy source code, column schema, row preview, code references, and execution duration).

• Complete data lineage: Automatically resolve Malloy source dependencies — including joined sources — to build a complete asset graph visible in the Dagster UI.

• Materialization: Queries are compiled and executed via malloy-cli or the malloy Python SDK. Results are surfaced as Apache Arrow, enabling zero-copy handoff to downstream assets.

• Data quality checks: Write validation queries directly in Malloy and have them run automatically as Dagster asset checks. Failed checks block downstream materializations, appear in the Dagster UI timeline, and are tracked in the asset health history — without any extra orchestration code.

  # Verify Customer IDs are non-null
  query: check_valid_customer_ids is orders -> {
    where: customer_id is null
    aggregate: invalid_count is count()
  }
  

Quickstart

Try dagster-malloy using:

uvx dagster-malloy-demo

This generates a sample project (./malloy_demo) and launches the Dagster UI at http://127.0.0.1:3000.

Installation

uv add dagster-malloy

Usage

1. Loading Malloy assets

Use load_malloy_assets to discover and construct Dagster assets from .malloy files or .malloynb notebooks in a directory:

from pathlib import Path
from dagster import Definitions
from dagster_malloy import load_malloy_assets, MalloyResource

# Loads Malloy assets using pre-compiled manifest (if present) or dynamic Node.js parsing
malloy_assets = load_malloy_assets(
    path=Path(__file__).parent / "models",
    use_manifest_if_exists=True,  # Default: True
)

defs = Definitions(
    assets=[malloy_assets],
    resources={
        "malloy": MalloyResource(
            cli_path="npx malloy-cli",
        ),
    },
)

2. AST Manifests & Serverless / Python-Only Deployments

In production or serverless environments (Cloud Run, ECS, Kubernetes), you can eliminate 100% of the Node.js runtime dependency for loading Dagster asset definitions by pre-compiling an AST manifest during CI/CD or Docker build.

Building the Manifest (CI/CD / Dockerfile):

Use the dagster-malloy build-manifest CLI command:

# Pre-compile Malloy AST metadata into analytics/malloy_manifest.json
dagster-malloy build-manifest analytics/ --output analytics/malloy_manifest.json

Loading Pre-compiled Manifests:

When malloy_manifest.json exists alongside your models (or when manifest_path is explicitly passed to load_malloy_assets), dagster-malloy loads asset definitions in pure Python (< 1ms) without calling Node.js.

malloy_assets = load_malloy_assets(
    path=PROJECT_ROOT / "analytics",
    manifest_path=PROJECT_ROOT / "analytics" / "malloy_manifest.json",  # Optional explicit path
    use_manifest_if_exists=True,  # Default: True
)

If Node.js is missing from $PATH and no manifest is available, dagster-malloy raises an explicit MalloyEnvironmentError with actionable instructions.

3. Execution configuration

dagster-malloy compiles and executes queries using malloy-cli / npx malloy-cli via MalloyResource:

resource = MalloyResource(
    cli_path="npx malloy-cli",  # Optional custom executable or path
    config_path="path/to/malloy-config.json",  # Optional path to database connections config
    project_dir="path/to/project",  # Optional project root for relative file paths
)

4. Custom translator (MalloyTranslator)

Subclass MalloyTranslator to customize asset keys, tags, group names, metadata, or upstream dependencies:

from dagster import AssetKey
from dagster_malloy import MalloyTranslator, MalloyTranslatorData, load_malloy_assets


class CustomMalloyTranslator(MalloyTranslator):
    def get_asset_key(self, data: MalloyTranslatorData) -> AssetKey:
        return AssetKey(["analytics", data.query_info.name])

    def get_group_name(self, data: MalloyTranslatorData) -> str:
        return "malloy_models"


malloy_assets = load_malloy_assets(
    path="./models",
    translator=CustomMalloyTranslator(),
)

5. Data quality checks

Use build_malloy_asset_checks to discover check queries in a .malloy file and register them as Dagster AssetCheckResult checks attached to a target asset:

from dagster import AssetKey
from dagster_malloy import build_malloy_asset_checks

checks = build_malloy_asset_checks(
    file_path="./models/sales.malloy",
    target_asset_key=AssetKey(["sales", "customer_analytics"]),
)

A query is recognised as a check if it's name starts with check_, test_, assert_ (eg. query: check_valid_ids is ...) or if it's annotated with # @check, # @test or # @assert before the query definition.

A check passes when the query returns zero rows, or when the first row contains invalid_count = 0 or fail_count = 0.

Example Project

A self-contained runnable example project is available in dagster_malloy_demo with instructions to run locally.

To clone and run the example locally:

git clone https://github.com/mathisdrn/dagster-malloy.git
cd dagster-malloy/dagster_malloy_demo
uv run generate_data.py
uv run dg dev -f definitions.py

Open http://127.0.0.1:3000 to view the asset catalog and lineage graph.

Contributing

Contributions, issues, and pull requests are welcome! Feel free to open an issue or submit a pull request on GitHub.