dagster-dataframely

Dataframely validates Polars frames against a schema. Dagster has two built-in places to show a schema: the Columns tab and asset checks. dagster-dataframely attaches a Dataframely schema to a Dagster asset, so you declare a table once and Dagster shows it in both places.

import dataframely as dy
import polars as pl

import dagster_dataframely as dd


class CustomersSchema(dy.Schema):
    """A storefront customer account."""

    customer_id = dy.String(primary_key=True, description="Account identifier.")
    email = dy.String(nullable=False, description="Address for receipts.")
    lifetime_value = dy.Float64(nullable=False, min=0.0, description="Spend to date.")


@dd.asset(CustomersSchema)
def customers(raw_customers: pl.DataFrame) -> pl.DataFrame:
    """Validate the storefront's customer accounts."""
    return raw_customers

That is the whole integration. From one declaration you get the catalog's Columns tab, one asset check per rule, and a code version derived from the schema. A row that fails an "error" rule fails the run. With filter=True, the run writes that row to a separate table instead and writes the other rows to the asset's table. Get started covers each of those, with screenshots of the demo pipeline.

Installation

uv add dagster-dataframely

You need Python 3.12 or newer. The dependencies are dagster, dataframely and polars, plus universal-pathlib, which dagster already installs.

This package includes an IO manager only for its examples, dd.data.ParquetIOManager. In your own project, bind a library one. dagster-polars writes Polars frames to a filesystem or object store, and dagster-duckdb-polars writes them to a warehouse. Any IO manager that stores by asset key works.

Pre-1.0. A characterization test covers the public surface, so it never changes by accident. It can still change: a 0.x minor release can include breaking changes. Upgrading has the migration for each one.

Documentation

https://ozanozbeker.com/dagster-dataframely has the guide, the reference and the changelog. Contributor documentation stays in the repository: CONTEXT.md is the glossary, and docs/pre-1.0.md records the decisions and the measurements the package was built on.