Skip to main content

The data orchestration platform built for productivity.

Project description

dagster logo

Remember to star the Dagster GitHub repo for future reference.

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Here is an example of a graph of three assets defined in Python:

from dagster import asset
from pandas import DataFrame, read_html, get_dummies
from sklearn.linear_model import LinearRegression

@asset
def country_populations() -> DataFrame:
    df = read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"]
    df["change"] = df["change"].str.rstrip("%").str.replace("−", "-").astype("float")
    return df

@asset
def continent_change_model(country_populations: DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(get_dummies(data[["continent"]]), data["change"])

@asset
def continent_stats(country_populations: DataFrame, continent_change_model: LinearRegression) -> DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

The graph loaded into Dagster's web UI:

An example asset graph as rendered in the Dagster UI

Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.

Quick Start:

If you're new to Dagster, we recommend reading about its core concepts or learning with the hands-on tutorial.

Dagster is available on PyPI and officially supports Python 3.7+.

pip install dagster dagit

This installs two modules:

  • Dagster: The core programming model.
  • Dagit: The web interface for developing and operating Dagster jobs and assets.

Running on Using a Mac with an M1 or M2 chip? Check the install details here.

Documentation

You can find the full Dagster documentation here, including the 'getting started' guide.


Key Features:

image

Dagster as a productivity platform

Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.

Dagster as a robust orchestration engine

Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.

Dagster as a unified control plane

Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.


Master the Modern Data Stack with integrations

Dagster provides a growing library of integrations for today’s most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.


image

Community

Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.

Join our community here:

Contributing

For details on contributing or running the project for development, check out our contributing guide.

License

Dagster is Apache 2.0 licensed.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster-1.3.0.tar.gz (975.5 kB view details)

Uploaded Source

Built Distribution

dagster-1.3.0-py3-none-any.whl (1.2 MB view details)

Uploaded Python 3

File details

Details for the file dagster-1.3.0.tar.gz.

File metadata

  • Download URL: dagster-1.3.0.tar.gz
  • Upload date:
  • Size: 975.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.9.6 requests/2.28.2 setuptools/65.5.1 requests-toolbelt/0.10.1 tqdm/4.65.0 CPython/3.10.11

File hashes

Hashes for dagster-1.3.0.tar.gz
Algorithm Hash digest
SHA256 aae48411b9ce8802510837209861746c030dd62d34c7a5de9249d17bbc680300
MD5 6bb47d4e68788261a3b17cc864f11013
BLAKE2b-256 11970522dffd2e57f656818d9420f85e6b038f0a3a4da73877ac8eda763d5950

See more details on using hashes here.

File details

Details for the file dagster-1.3.0-py3-none-any.whl.

File metadata

  • Download URL: dagster-1.3.0-py3-none-any.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.9.6 requests/2.28.2 setuptools/65.5.1 requests-toolbelt/0.10.1 tqdm/4.65.0 CPython/3.10.11

File hashes

Hashes for dagster-1.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9651c38cf59f0999459fde969280b323e7bef2e143d6e0c9eb9fa7d071808173
MD5 8d666b803310bef76c67a3aebeb71a7e
BLAKE2b-256 22dbf1936e66a3019ea44f7047be4fef4ebce2d60dc404f4ce650484197144c3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page