Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.
It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.
With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.
Here is an example of a graph of three assets defined in Python:
import dagster as dg
import pandas as pd
from sklearn.linear_model import LinearRegression
@dg.asset
def country_populations() -> pd.DataFrame:
df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
df["change"] = df["change"].str.rstrip("%").astype("float")
return df
@dg.asset
def continent_change_model(country_populations: pd.DataFrame) -> LinearRegression:
data = country_populations.dropna(subset=["change"])
return LinearRegression().fit(pd.get_dummies(data[["continent"]]), data["change"])
@dg.asset
def continent_stats(country_populations: pd.DataFrame, continent_change_model: LinearRegression) -> pd.DataFrame:
result = country_populations.groupby("continent").sum()
result["pop_change_factor"] = continent_change_model.coef_
return result
The graph loaded into Dagster's web UI:
Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.
Quick Start:
If you're new to Dagster, we recommend checking out the docs or following the hands-on tutorial.
Dagster is available on PyPI and officially supports Python 3.9 through Python 3.14.
uv add dagster dagster-webserver dagster-dg-cli
Documentation
You can find the full Dagster documentation here, including the Quickstart guide.
Key Features:
Dagster as a productivity platform
Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.
Dagster as a robust orchestration engine
Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.
Dagster as a unified control plane
Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.
Master the Modern Data Stack with integrations
Dagster provides a growing library of integrations for today’s most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.
Community
Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.
Join our community here:
- 🌟 Star us on GitHub
- 📥 Subscribe to our Newsletter
- 🐦 Follow us on Twitter
- 🕴️ Follow us on LinkedIn
- 📺 Subscribe to our YouTube channel
- 📚 Read our blog posts
- 👋 Join us on Slack
- 🗃 Browse Slack archives
- ✏️ Start a GitHub Discussion
Contributing
For details on contributing or running the project for development, check out our contributing guide.
License
Dagster is Apache 2.0 licensed.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dagster-1.13.12.tar.gz.
File metadata
- Download URL: dagster-1.13.12.tar.gz
- Upload date:
- Size: 3.6 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8397840e221cefc2125b0b2f27e492bd5a26584218e764c7168cd7855c56fb07
|
|
| MD5 |
4c0c54e36b8c7b4018413284a48e1c92
|
|
| BLAKE2b-256 |
bad1ed249019f5a5b09e3eac6200a2b921c47f5074c54cc560b20e0de1fa21df
|
File details
Details for the file dagster-1.13.12-py3-none-any.whl.
File metadata
- Download URL: dagster-1.13.12-py3-none-any.whl
- Upload date:
- Size: 2.0 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f271c6dfcf2f30298c110179d215268e71995928491d72ded3a9095c84868a70
|
|
| MD5 |
e6295032c2613590786b210a0fd3346a
|
|
| BLAKE2b-256 |
52a6a561ae62bd69ac7e47b30ab5dbf99a31130c2bbca00e56fbad43870bde8b
|
