Skip to main content

The data orchestration platform built for productivity.

Project description

dagster logo


Dagster

Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.

You declare functions that you want to run and the data assets that those functions produce or update. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.

If you're new to Dagster, we recommend reading about its core concepts or learning with the hands-on tutorial.

An asset graph defined in Python:

from dagster import asset
from pandas import DataFrame, read_html, get_dummies
from sklearn.linear_model import LinearRegression

@asset
def country_populations() -> DataFrame:
    df = read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "continent", "rg", "pop2018", "pop2019", "change"]
    df["change"] = df["change"].str.rstrip("%").str.replace("−", "-").astype("float")
    return df

@asset
def continent_change_model(country_populations: DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(
        get_dummies(data[["continent"]]), data["change"]
    )

@asset
def continent_stats(
    country_populations: DataFrame, continent_change_model: LinearRegression
) -> DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

The graph loaded into Dagster's web UI:

image

Installation

Dagster is available on PyPI and officially supports Python 3.7+.

pip install dagster dagit

This installs two modules:

  • Dagster: The core programming model.
  • Dagit: The web interface for developing and operating Dagster jobs and assets.

Documentation

You can find the full Dagster documentation here.

Community

Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.

Join our community here:

Contributing

For details on contributing or running the project for development, check out our contributing guide.

License

Dagster is Apache 2.0 licensed.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster-1.0.16.tar.gz (823.3 kB view details)

Uploaded Source

Built Distribution

dagster-1.0.16-py3-none-any.whl (1.0 MB view details)

Uploaded Python 3

File details

Details for the file dagster-1.0.16.tar.gz.

File metadata

  • Download URL: dagster-1.0.16.tar.gz
  • Upload date:
  • Size: 823.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.8.3 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.12

File hashes

Hashes for dagster-1.0.16.tar.gz
Algorithm Hash digest
SHA256 dee40a8de634966de49bfdf4dd26f690e1c03c6fb1cc00977733eb7cb58ea3bf
MD5 c29f640ef3f5c5b8f95a9a89f1a9f00d
BLAKE2b-256 63f2bbb2278636ed7627807806188ffc3978cac65840559b065ed9cad9b41c5a

See more details on using hashes here.

File details

Details for the file dagster-1.0.16-py3-none-any.whl.

File metadata

  • Download URL: dagster-1.0.16-py3-none-any.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.8.3 requests/2.28.1 setuptools/57.5.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/3.8.12

File hashes

Hashes for dagster-1.0.16-py3-none-any.whl
Algorithm Hash digest
SHA256 239c84a256da3109a9f3b7af0adb53dcefb8c9e4901331657771706a5a7e17b6
MD5 0f86a0f3c7f068da9fd1b11b421e66e2
BLAKE2b-256 1f4c25f80ae0a5b8ce816d61004f6cb67e12e34a8dd202f385076018c25598e5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page