dagster-dex

Let dex read your Dagster asset graph the way it reads a dbt project: keys, joins, sources, semantics, drift.

dex reports where a warehouse has drifted from the structure a project declares. It reads dbt projects natively. This package makes a Dagster asset graph readable the same way, without the graph having to pretend to be a dbt project first.

Copyright 2026 David Anaya. Apache-2.0.

pip install dagster-dex          # the core: no orchestrator, no engine
pip install 'dagster-dex[dex]'   # with the dex-core boundary

Or from source, for main ahead of the last release:

pip install 'git+https://github.com/catincloud-labs/dagster-dex'
pip install 'dagster-dex[dex] @ git+https://github.com/catincloud-labs/dagster-dex'

Python 3.10+, matching Dagster's own floor. The [dex] extra needs 3.11+, because the engine does: on 3.10 you can still reduce a graph and write an artifact for a machine that has the engine to read it.

What it finds

examples/walk_the_whole_loop.py puts a small Dagster asset graph in front of a local DuckDB warehouse, breaks the warehouse and one of the project's definitions by hand, and lets dex find the drift through this package. It needs no cloud account, no credential and nothing private. From the root of a clone of this repository:

uv run --no-project --with-editable . --with 'dagster>=1.13' \
  --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 --with duckdb \
  python examples/walk_the_whole_loop.py

Some of the lines it prints:

leg 5 ok          : 2 finding(s); dim_date impacts ['dim_date', 'fact_sales']
leg 7/8 ok        : wrote it, and the project gained [('dim_date', 'date_key')]
leg 11 ok         : dangling_source on raw.sales_events, declared_in sources/fact_sales.yml
leg 12 ok         : definition_changed, and sales.amount reported as uncheckable

• Leg 5. One duplicate row in each of dim_date and fact_sales breaks both keys, hence two findings, and the dim_date finding names fact_sales among the models it impacts. Nothing in the warehouse connects those two tables; the Dagster asset graph does.
• Legs 7 and 8. The unique test dex proposed for the broken key is written into the hand-written declaration YAML through this package, and reading the project back shows the key declared.
• Leg 11. A source table the project declares is dropped, and the finding names the file that declared it.
• Leg 12. A measure is redefined as amount * 1.2. dex reports definition_changed, which no inspection of the warehouse finds because nothing in the warehouse changed, and says the new expression no longer maps to a column it can check, rather than passing it.

CI runs this example on every pull request and refuses the change if any line quoted above is not one the run printed.

Pointing dex at your graph

Once the package is installed, name the format in dex's own config:

# .dex/config.yml
project:
  format: dagster
  options:
    assets: my_project.definitions:all_assets

What this is

A working sketch of a contract, not a finished integration. It exists to make a design argument concrete enough to disagree with: an asset graph is a perfectly good source of project truth, and reading one does not require it to pretend to be a dbt project first.

One boundary, stated up front: this format describes one orchestrated asset graph. It does not describe, and will not absorb, a sibling project in another format - two formats each describing the same warehouse table is exactly the duplication this package's design argument rejects. Where two projects meet, the warehouse is the join: each describes what it builds, and lineage across them is read from the tables, not from either project.

The seam, in three tiers

Which tier you get depends on how the project was built, and that is the point. DagsterProject reaches tier 2. EditableDagsterProject reaches tier 3, and the factory returns one only when the project points at a declarations directory on disk - somewhere an edit can land. Read one from an artifact: alone and you get tier 2, because an artifact is a JSON file with no directory behind it.

The graph is no longer the only way to have one. A declarations: directory named beside an artifact: reaches tier 3 as well; see Two ways to name the project for what that directory supersedes and why it has to.

The split is a class, not a flag, and it has to be. EditableProject is runtime_checkable, so it matches on the method being present: put propose_edits on the shared class and every project claims the write tier and then refuses at call time. isinstance(project, EditableProject) is the truth for each instance, so a caller finds out by asking rather than by receiving an empty result that looks like success.

What can receive an edit is the declarations, never the models. The models are a reduction of a running asset graph, whose source of truth is the code that built it; the declared keys and joins are hand-written YAML that nothing regenerates. Those are different claims about different files.

Four model decisions worth arguing about

A key is always a tuple of columns. A single-column grain is the n == 1 case, not a different type. Modelling it the other way is what makes a composite grain inexpressible until someone widens the type - and after widening, every reader handles two shapes forever.

There is no project directory. A path is one format's way of locating itself, not a property every project has.

Freshness is a tri-state. FRESH / STALE / NOT_APPLICABLE. A boolean cannot tell "current" from "there is no such artifact here", so a format with nothing to compile reports itself identically to one that just compiled.

A required field is a refusal to invent. ExternalSource requires a source system because the consumer requires one; making it optional would only move the moment of invention into the boundary code, where a fabricated value cannot be told from a declared one. Where the consumer requires something a project may genuinely lack, such as a file path, the field is optional here and the loss is reported instead of filled in.

External sources: the edge a graph cannot see

Everything in models is something the project builds. ExternalSource is where it stops and depends on a warehouse object somebody else keeps honoring.

These cannot be derived from the asset graph. A model that reads a table in its own SQL, because the source system offers no other interface, draws no dependency edge, so the read is invisible to any amount of traversal. It is genuinely extra information and has to be declared, keyed by the model that reads it. A format that inferred sources from dependencies would find none and look like it had checked.

They are not decorative on the consumer side either: they drive a dangling_source finding and the set of tables a project is understood to cover. A format that reports none silently narrows its own blast radius.

Layout, and why the dependencies sit where they do

model.py         the neutral model          stdlib only
protocol.py      the tiered seam            stdlib only
declarations.py  input parsing              + pyyaml
project.py       the reduction, the writer  stdlib only
conformance.py   the importable contract    + pytest
dex.py           the dex-core boundary      + exmergo-dex-core, lazily

The engine coupling is one file. Everything else, including the conformance suite and the whole write path, runs with dex-core uninstalled, which is what keeps the design from being shaped by whatever the engine happens to look like today. dex.py imports public API only; tests/test_dex_bridge.py asserts that with an AST walk rather than trusting it.

The write path is in project.py rather than at the boundary for that reason. The conflict handshake is the one part of this package where getting it wrong costs somebody their work rather than an inaccurate report, so it is asserted in the step that runs with no engine installed - tests/test_write_tier.py.

Running it

From the repository root:

# the core: no orchestrator, no engine
uv run --no-project --with-editable . \
    --with pytest==8.4.1 \
    python -m pytest tests \
    --ignore=tests/test_dex_bridge.py \
    --ignore=tests/test_upstream_contract.py

# with the engine, to exercise the boundary
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 \
    python -m pytest tests \
    --ignore=tests/test_upstream_contract.py

--with-editable is not optional: without it the package is never installed and every test errors at collection on No module named 'dagster_dex'. Both run in CI, and the first is a control: it is what holds the engine coupling to one file, so an exmergo_dex_core import anywhere else turns it red at collection.

The engine is pinned to exmergo-dex-core==1.8.0 here and in CI, which is not what the [dex] extra publishes. The extra is exmergo-dex-core~=1.8, so consumers are not forced onto one release; the exact pin is where the demonstration lives, because a claim about what passed should name the version it passed against.

Against dex-core's own contract

tests/test_upstream_contract.py runs this format against the conformance suite dex-core ships, which is upstream's acceptance criterion for a second project format (exmergo/dex#144). That suite ships in v1.5.2, released 2026-08-05, via exmergo/dex#192 - our own PR, so the criterion is now judged by upstream's released code rather than by a branch:

DEX_UPSTREAM_CONTRACT_REQUIRED=1 \
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 \
    python -m pytest tests/test_upstream_contract.py

DEX_UPSTREAM_CONTRACT_REQUIRED=1 is what makes it a signal, and the release did not change that. Without the variable the file skips itself when the contract is unimportable, which is right at a developer's desk and wrong in CI, where an unrun file and a passing file look the same. With it, a missing contract is a collection error - which still matters against a release, because a yank or a bad bump lands in exactly the same place a rewritten branch used to.

Using the contract on another format

from dagster_dex.conformance import FingerprintedProjectContract

class TestMyFormat(FingerprintedProjectContract):
    def make_project(self, declarations, semantics, sources):
        return MyProject(declaration_sources=declarations,
                         semantic_sources=semantics,
                         source_declarations=sources)

The assertions that matter most are behavioural: declarations() must not raise on an absent, empty, or malformed project. That is the property a second implementation is most likely to get wrong, and a suite checking only shapes will pass it.

make_project took two arguments before external sources landed. The third is a breaking change to this contract, taken deliberately while the suite has no outside implementers rather than carried forever as an optional hook that could be skipped - and a skipped assertion is not a passing one.

Status

Alpha, and the contract may move before 1.0. The tiered seam is a proposal, not a settled interface: it exists to be argued with, and the argument may change it. Pin the minor if you depend on it.

CHANGELOG.md names what changed between releases and which changes were breaking, which is the question this section cannot answer for a consumer already on an older version.

dex finds this format by name. From dex-core 1.6.0 the engine resolves the exmergo_dex_core.projects entry-point group, which this package registers in, so a format can be named from .dex/config.yml, from a dotted mypkg.projects:my_project path, or from that group, with --project-format overriding on the CLI. The entry point names dex:project_from_context, which takes a context and returns a DexProject, and a regression test asserts the registration and that it loads to the callable. How it came to be registered that way is in the changelog.

Every row runs from a clean checkout with no access to anything private: CONTRIBUTING.md gives the commands, and the workflows under .github/workflows/ run the rest.

Two ways to name the project, and the reason there are two

assets: reduces a live asset graph in the calling process. That is the honest form, and its cost is dominated by importing the code location rather than by the reduction - which is not something laziness or caching can reach, because a host that builds a project per command never holds one long enough to amortize it. How long that import takes is a property of your project rather than of this package, so it is the thing to measure before choosing between the two forms: time dagster definitions list against your own code location and you have the number, on your machine, where it is true.

artifact: is the answer for a host that cannot pay that: the side that already has the graph reduces it once and writes the result down, and the side that answers requests reads it back.

project:
  format: dagster
  options:
    artifact: project/my_project.json   # written by dagster_dex.artifact.dump

Exactly one of the two is required, and naming both is refused rather than resolved: a snapshot and a live graph disagree by design. An artifact carries its own semantics, its own sources and its own name, so semantics:, sources: and name: are refused beside it rather than silently ignored - each would modify nothing while reading as though the files on disk were live, which is the ordinary shape of a config quietly a day out of date.

A missing artifact is refused, not read as an empty project. An empty project is a valid one, so the tolerant reading would report a broken deploy as a warehouse with nothing declared, quietly, and for as long as it lasted.

Reaching the write tier over the artifact transport

declarations: is the one option admitted beside artifact:, and it supersedes the declaration text the artifact carries:

project:
  format: dagster
  options:
    artifact: project/my_project.json
    declarations: declared            # the live source, and where edits land

The obvious reading of that option - the artifact still declares, and the directory only says where an edit may land - cannot be built, and the reason is worth knowing before proposing it again:

• An artifact keys its declarations by a bare stem, and must keep doing so. It has no directory to have come from, and inventing one would be fabricated provenance.
• An editing surface is checked segment-wise, so declarations admits declarations/dim_date.yml and refuses a bare dim_date. A bare stem is inside no surface.
• So an edit view built from the artifact's text is empty. Every edit pins against a file it believes absent, and every apply is then a conflict on a file that plainly exists - refused on every write, forever, which is worse than declining the tier.

The directory has to be read, so it is the declarations. That costs nothing the artifact was protecting: artifact: exists because reducing a live graph imports a code location, and hand-written YAML needs neither an import nor an orchestrator. What the artifact still answers for is the graph.

Because the artifact's copy of that text loses, notes() says so - a mapping that drops something without disclosing it is the failure notes exists to prevent. A missing directory is refused rather than read as "nothing is editable": a declined write tier is indistinguishable from a format that never had one, so a typo would be silent.

An empty directory is accepted, and the asymmetry is a contract rather than an accident. A consumer's provisioning order can legitimately create the directory before its first content arrives, so first boot depends on empty being admitted; the project reaches the write tier with an empty editing view, and the view fills as files do. A missing directory means a wrong path; an empty one means a true state. Tightening the empty arm into a refusal would be a breaking change to that provisioning order, not a tidier default.

The directory's parent becomes the project root, and that derivation is a guarantee (stable at least until 1.0). Keys are <directory-name>/<file>, relative to the parent, and downstream tooling stores plans relative to that root - so a plan made against one checkout applies against another only because both derive the same root from the same configured path. A change here relocates every stored plan's project_dir; it is pinned by a test in this repository so the rule goes red where it lives.

Writing one, from the side that has the graph

dagster_dex.artifact.dump writes the file. It is deliberately orchestrator-free - the reduction reads asset definitions structurally - so the scheduling around it is yours. This is what that looks like in Dagster, and it is user code: copy it, do not import it.

from datetime import datetime, timezone

import dagster as dg
from dagster_dex import DagsterProject
from dagster_dex.artifact import dump

@dg.op
def write_project_artifact(context) -> int:
    from my_project.definitions import all_assets   # deferred: see below

    project = DagsterProject.from_asset_graph(all_assets, name="my_project")
    declarations = project.declarations()
    dump(
        "/shared/project/my_project.json",
        name="my_project",
        models=declarations.models,
        generated_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        declaration_sources=my_declaration_yaml,      # {filename: text}
    )
    context.log.info("wrote %s models", len(declarations.models))
    return len(declarations.models)

@dg.job
def write_project_artifact_job() -> None:
    write_project_artifact()

write_project_artifact_schedule = dg.ScheduleDefinition(
    job=write_project_artifact_job,
    cron_schedule="0 6 * * *",
    execution_timezone="UTC",
    default_status=dg.DefaultScheduleStatus.RUNNING,
)

Four things in that sketch are not incidental:

• The asset import is deferred into the op body. A module-scope import of the graph is a cycle if your definitions.py also imports this module, and a failure in it takes down the whole code location rather than one job.
• generated_at is yours to supply. The library has no clock on purpose: a default would make every artifact look freshly written, including one produced by a replay.
• The path's parent must already exist. dump will not create it - a missing parent is a configuration mistake, and creating it writes the artifact somewhere nobody reads.
• Schedule it. An artifact that regenerates when someone remembers is a stale artifact, and a stale project is a wrong drift report rather than an absent one.

The sketch's mechanics are executed by CI - examples/write_project_artifact.py runs the same op body against a real graph, executes the job in process, reads the artifact back, and constructs the schedule, so the code above cannot silently rot. What CI deliberately does not decide is the schedule values: scheduling is a decision, and a library that made it for you would be wrong in every deployment that differs - which is why the sketch stays here, with placeholders, as user code.

Verifying a release

Each file on PyPI carries a PEP 740 attestation, which PyPI checks and shows: it names this repository's publish.yml, run at the release's v* tag. From the first release after v0.6.2, that tag is also signed by the maintainer, so the chain ends at a public key rather than at repository settings you cannot read. v0.6.2 and every tag before it are unsigned and stay that way: a signature added now would vouch for them after the fact, not at their release.

To check one, with git 2.34 or newer and OpenSSH 8.8 or newer, put the version you installed in place of vX.Y.Z:

git clone https://github.com/catincloud-labs/dagster-dex
cd dagster-dex
cat .github/allowed_signers
curl -s https://api.github.com/users/catincloudlabs/ssh_signing_keys
git -c gpg.ssh.allowedSignersFile=.github/allowed_signers tag -v vX.Y.Z

• The key comes before the file. Every key in .github/allowed_signers has to be one GitHub lists as a signing key of the maintainer's account, catincloudlabs, which is what curl prints. The file travels in the same repository as the tag it vouches for, so on its own it proves nothing.
• Then the tag. tag -v prints the tag, then Good "git" signature for the principal in that file, and exits 0. Anything else (no signature found, No principal matched) means the tag is not one the maintainer signed.
• Then two lines of what it printed. tag has to name the version you asked for: a signature covers the name written inside the tag, so a tag signed under another name still verifies wherever it is moved. object is the commit the release was built from, the same commit its PyPI attestation names.
• The release workflow runs the same checks first. Its build job verifies the tag, and the name inside it, before it runs a test or builds anything, and a tag that fails stops the release before anything is uploaded.