dagster-dex

A Dagster asset graph read as a project format for dex, and the tiered seam it proposes for reading one.

Copyright 2026 David Anaya. Apache-2.0.

pip install dagster-dex          # the core: no orchestrator, no engine
pip install 'dagster-dex[dex]'   # with the dex-core boundary

Or from source, for main ahead of the last release:

pip install 'git+https://github.com/catincloud-labs/dagster-dex'
pip install 'dagster-dex[dex] @ git+https://github.com/catincloud-labs/dagster-dex'

Python 3.10+, matching Dagster's own floor. The [dex] extra needs 3.11+, because the engine does: on 3.10 you can still reduce a graph and write an artifact for a machine that has the engine to read it.

Once installed, dex can name this format directly:

# .dex/config.yml
project:
  format: dagster
  options:
    assets: my_project.definitions:all_assets

dex reads a project's declared structure (keys, joins, sources, semantics) and reports where the warehouse has drifted from it. It reads dbt projects natively. This package makes a Dagster asset graph readable the same way, without the graph having to pretend to be a dbt project first.

What this is

A working sketch of a contract, not a finished integration. It exists to make a design argument concrete enough to disagree with: an asset graph is a perfectly good source of project truth, and reading one does not require it to pretend to be a dbt project first.

The seam, in three tiers

Which tier you get depends on how the project was built, and that is the point. DagsterProject reaches tier 2. EditableDagsterProject reaches tier 3, and the factory returns one only when the project points at a declarations directory on disk - somewhere an edit can land. Read one from an artifact: alone and you get tier 2, because an artifact is a JSON file with no directory behind it.

The graph is no longer the only way to have one. A declarations: directory named beside an artifact: reaches tier 3 as well; see Two ways to name the project for what that directory supersedes and why it has to.

The split is a class, not a flag, and it has to be. EditableProject is runtime_checkable, so it matches on the method being present: put propose_edits on the shared class and every project claims the write tier and then refuses at call time. isinstance(project, EditableProject) is the truth for each instance, so a caller finds out by asking rather than by receiving an empty result that looks like success.

What can receive an edit is the declarations, never the models. The models are a reduction of a running asset graph, whose source of truth is the code that built it; the declared keys and joins are hand-written YAML that nothing regenerates. Those are different claims about different files.

Four model decisions worth arguing about

A key is always a tuple of columns. A single-column grain is the n == 1 case, not a different type. Modelling it the other way is what makes a composite grain inexpressible until someone widens the type - and after widening, every reader handles two shapes forever.

There is no project directory. A path is one format's way of locating itself, not a property every project has.

Freshness is a tri-state. FRESH / STALE / NOT_APPLICABLE. A boolean cannot tell "current" from "there is no such artifact here", so a format with nothing to compile reports itself identically to one that just compiled.

A required field is a refusal to invent. ExternalSource requires a source system because the consumer requires one; making it optional would only move the moment of invention into the boundary code, where a fabricated value cannot be told from a declared one. Where the consumer requires something a project may genuinely lack, such as a file path, the field is optional here and the loss is reported instead of filled in.

External sources: the edge a graph cannot see

Everything in models is something the project builds. ExternalSource is where it stops and depends on a warehouse object somebody else keeps honoring.

These cannot be derived from the asset graph. A model that reads a table in its own SQL, because the source system offers no other interface, draws no dependency edge, so the read is invisible to any amount of traversal. It is genuinely extra information and has to be declared, keyed by the model that reads it. A format that inferred sources from dependencies would find none and look like it had checked.

They are not decorative on the consumer side either: they drive a dangling_source finding and the set of tables a project is understood to cover. A format that reports none silently narrows its own blast radius.

Layout, and why the dependencies sit where they do

model.py         the neutral model          stdlib only
protocol.py      the tiered seam            stdlib only
declarations.py  input parsing              + pyyaml
project.py       the reduction, the writer  stdlib only
conformance.py   the importable contract    + pytest
dex.py           the dex-core boundary      + exmergo-dex-core, lazily

The engine coupling is one file. Everything else, including the conformance suite and the whole write path, runs with dex-core uninstalled, which is what keeps the design from being shaped by whatever the engine happens to look like today. dex.py imports public API only; tests/test_dex_bridge.py asserts that with an AST walk rather than trusting it.

The write path is in project.py rather than at the boundary for that reason. The conflict handshake is the one part of this package where getting it wrong costs somebody their work rather than an inaccurate report, so it is asserted in the step that runs with no engine installed - tests/test_write_tier.py.

Running it

From the repository root:

# the core: no orchestrator, no engine
uv run --no-project --with-editable . \
    --with pytest==8.4.1 \
    python -m pytest tests \
    --ignore=tests/test_dex_bridge.py \
    --ignore=tests/test_upstream_contract.py

# with the engine, to exercise the boundary
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.6.6 --with sqlglot==30.13.0 \
    python -m pytest tests \
    --ignore=tests/test_upstream_contract.py

--with-editable is not optional: without it the package is never installed and every test errors at collection on No module named 'dagster_dex'. These are the two commands CI runs, and the first is a control: it is what holds the engine coupling to one file, so an exmergo_dex_core import anywhere else turns it red at collection.

The engine is pinned to exmergo-dex-core==1.6.6 here and in CI, which is not what the [dex] extra publishes. The extra is exmergo-dex-core~=1.6.4, so consumers are not forced onto one patch release; the exact pin is where the demonstration lives, because a claim about what passed should name the version it passed against.

Against dex-core's own contract

tests/test_upstream_contract.py runs this format against the conformance suite dex-core ships, which is upstream's acceptance criterion for a second project format (exmergo/dex#144). That suite ships in v1.5.2, released 2026-08-05, via exmergo/dex#192 - our own PR, so the criterion is now judged by upstream's released code rather than by a branch:

DEX_UPSTREAM_CONTRACT_REQUIRED=1 \
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.6.6 --with sqlglot==30.13.0 \
    python -m pytest tests/test_upstream_contract.py

DEX_UPSTREAM_CONTRACT_REQUIRED=1 is what makes it a signal, and the release did not change that. Without the variable the file skips itself when the contract is unimportable, which is right at a developer's desk and wrong in CI, where an unrun file and a passing file look the same. With it, a missing contract is a collection error - which still matters against a release, because a yank or a bad bump lands in exactly the same place a rewritten branch used to.

Using the contract on another format

from dagster_dex.conformance import FingerprintedProjectContract

class TestMyFormat(FingerprintedProjectContract):
    def make_project(self, declarations, semantics, sources):
        return MyProject(declaration_sources=declarations,
                         semantic_sources=semantics,
                         source_declarations=sources)

The assertions that matter most are behavioural: declarations() must not raise on an absent, empty, or malformed project. That is the property a second implementation is most likely to get wrong, and a suite checking only shapes will pass it.

make_project took two arguments before external sources landed. The third is a breaking change to this contract, taken deliberately while the suite has no outside implementers rather than carried forever as an optional hook that could be skipped - and a skipped assertion is not a passing one.

Status

Alpha, and the contract may move before 1.0. The tiered seam is a proposal, not a settled interface: it exists to be argued with, and the argument may change it. Pin the minor if you depend on it.

CHANGELOG.md names what changed between releases and which changes were breaking, which is the question this section cannot answer for a consumer already on an older version.

The entry point stopped being inert on 2026-08-08. This section used to end: "nothing resolves it today, and an entry point nobody looks up is inert." dex-core 1.6.0 added resolution for exactly the exmergo_dex_core.projects group this package has declared since 0.1.0 - a format can now be named from .dex/config.yml, from a dotted mypkg.projects:my_project path, or from that entry-point group, with --project-format overriding on the CLI.

That arrived through exmergo/dex#171, which this package's own constraint shaped: a host reaching dex as a subprocess cannot hand an object in, so name resolution was the only door that worked.

And it was registered wrong. That paragraph originally ended "what is left between here and a resolvable format is packaging, not design", written before anyone had run it. Resolution found the entry point immediately and then refused: it named the class, and dex-core's ProjectFactory calls what it resolves with a ProjectContext, so the context bound to models. A second gap sat behind it: a bare DagsterProject is refused as "missing name, definitions", because this package says format/declarations() where the seam says name/definitions().

=> The lesson is worth more than the fix: a declared-but-unresolved extension point is not evidence that registration works. It was inert from 0.1.0, so there was no moment before 2026-08-08 at which it could have failed.

Both are fixed. The entry point names dex:project_from_context, which takes a context and returns a DexProject, and a regression test asserts the registration and that it loads to the callable.

This paragraph used to end with a hand-counted end-to-end result - a model count, a source count, semantic models and metrics, from dex driven as a subprocess against a private asset graph. It was true where it was written and it is unverifiable here: nobody outside can run it, and it was four numbers in a document, which is the thing this project's own rules forbid. What replaces it is smaller and checkable by anyone:

Everything above runs from a clean checkout with two commands and no access to anything private, which is the property the old sentence did not have.

Two ways to name the project, and the reason there are two

assets: reduces a live asset graph in the calling process. That is the honest form, and its cost is dominated by importing the code location rather than by the reduction - which is not something laziness or caching can reach, because a host that builds a project per command never holds one long enough to amortize it. How long that import takes is a property of your project rather than of this package, so it is the thing to measure before choosing between the two forms: time dagster definitions list against your own code location and you have the number, on your machine, where it is true.

This paragraph used to give a figure from a private project. It was measured, it was accurate where it was written, and nobody outside could reproduce it - the same defect as the four numbers removed above, left standing twenty lines below the paragraph that removed them.

artifact: is the answer for a host that cannot pay that: the side that already has the graph reduces it once and writes the result down, and the side that answers requests reads it back.

project:
  format: dagster
  options:
    artifact: project/my_project.json   # written by dagster_dex.artifact.dump

Exactly one of the two is required, and naming both is refused rather than resolved: a snapshot and a live graph disagree by design. An artifact carries its own semantics, its own sources and its own name, so semantics:, sources: and name: are refused beside it rather than silently ignored - each would modify nothing while reading as though the files on disk were live, which is the ordinary shape of a config quietly a day out of date.

A missing artifact is refused, not read as an empty project. An empty project is a valid one, so the tolerant reading would report a broken deploy as a warehouse with nothing declared, quietly, and for as long as it lasted.

Reaching the write tier over the artifact transport

declarations: is the one option admitted beside artifact:, and it supersedes the declaration text the artifact carries:

project:
  format: dagster
  options:
    artifact: project/my_project.json
    declarations: declared            # the live source, and where edits land

The obvious reading of that option - the artifact still declares, and the directory only says where an edit may land - cannot be built, and the reason is worth knowing before proposing it again:

• An artifact keys its declarations by a bare stem, and must keep doing so. It has no directory to have come from, and inventing one would be fabricated provenance.
• An editing surface is checked segment-wise, so declarations admits declarations/dim_date.yml and refuses a bare dim_date. A bare stem is inside no surface.
• So an edit view built from the artifact's text is empty. Every edit pins against a file it believes absent, and every apply is then a conflict on a file that plainly exists - refused on every write, forever, which is worse than declining the tier.

The directory has to be read, so it is the declarations. That costs nothing the artifact was protecting: artifact: exists because reducing a live graph imports a code location, and hand-written YAML needs neither an import nor an orchestrator. What the artifact still answers for is the graph.

Because the artifact's copy of that text loses, notes() says so - a mapping that drops something without disclosing it is the failure notes exists to prevent. A missing directory is refused rather than read as "nothing is editable": a declined write tier is indistinguishable from a format that never had one, so a typo would be silent.

Writing one, from the side that has the graph

dagster_dex.artifact.dump writes the file. It is deliberately orchestrator-free - the reduction reads asset definitions structurally - so the scheduling around it is yours. This is what that looks like in Dagster, and it is user code: copy it, do not import it.

from datetime import datetime, timezone

import dagster as dg
from dagster_dex import DagsterProject
from dagster_dex.artifact import dump

@dg.op
def write_project_artifact(context) -> int:
    from my_project.definitions import all_assets   # deferred: see below

    project = DagsterProject.from_asset_graph(all_assets, name="my_project")
    declarations = project.declarations()
    dump(
        "/shared/project/my_project.json",
        name="my_project",
        models=declarations.models,
        generated_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        declaration_sources=my_declaration_yaml,      # {filename: text}
    )
    context.log.info("wrote %s models", len(declarations.models))
    return len(declarations.models)

@dg.job
def write_project_artifact_job() -> None:
    write_project_artifact()

write_project_artifact_schedule = dg.ScheduleDefinition(
    job=write_project_artifact_job,
    cron_schedule="0 6 * * *",
    execution_timezone="UTC",
    default_status=dg.DefaultScheduleStatus.RUNNING,
)

Four things in that sketch are not incidental:

• The asset import is deferred into the op body. A module-scope import of the graph is a cycle if your definitions.py also imports this module, and a failure in it takes down the whole code location rather than one job.
• generated_at is yours to supply. The library has no clock on purpose: a default would make every artifact look freshly written, including one produced by a replay.
• The path's parent must already exist. dump will not create it - a missing parent is a configuration mistake, and creating it writes the artifact somewhere nobody reads.
• Schedule it. An artifact that regenerates when someone remembers is a stale artifact, and a stale project is a wrong drift report rather than an absent one.

This snippet is prose and nothing runs it, which is a real cost and is stated rather than hidden: examples/reduce_asset_graph.py is executed by CI and this is not. It is here because scheduling is a decision, and a library that made it for you would be wrong in every deployment that differs.