astronomer-cosmos 0.0.5

Apache Airflow Providers containing 3rd party integrations supported natively in Airflow

A framework for generating Apache Airflow DAGs from other workflows.

Installation

Install and update using pip:

pip install astronomer-cosmos

This only installs dependencies for core provider. To install all dependencies, run:

pip install 'astronomer-cosmos[all]'

To only install the dependencies for a specific integration, specify the integration name as extra argument, example to install dbt integration dependencies, run:

pip install 'astronomer-cosmos[dbt]'

Extras

Extra Name	Installation Command	Dependencies
all	pip install 'astronomer-cosmos[all]'	All
dbt	pip install 'astronomer-cosmos[dbt]'	dbt core

Example Usage

Imagine we have a dbt project located at ./dbt/my_project.

from cosmos.providers.dbt import DbtDag, DbtTaskGroup, DbtTask

# render as a DAG
dag = DbtDag(
    project_dir="./dbt/my_project",
    dag_id="my_dag",
    schedule_interval="@daily",
    default_args={"owner": "airflow"},
)

# render as a task group
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
    task_group = DbtTaskGroup(
        project_dir="./dbt/my_project",
        task_group_id="my_task_group",
    )

# render as an individual task
with DAG("my_dag", default_args={"owner": "airflow"}) as dag:
    task = DbtTask(
        project_dir="./dbt/my_project",
        task_id="my_task",
    )

Principles

Astronomer Cosmos provides a framework for generating Apache Airflow DAGs from other workflows. Every provider comes with two main components:

• extractors: These are responsible for extracting the workflow from the provider and converting it into Task and Group objects.

• operators: These are used when the workflow is converted into a DAG. They are responsible for executing the tasks in the workflow.

Astronomer Cosmos is not opinionated in the sense that it does not enforce any rendering method. Rather, it comes with the tools to render workflows as Airflow DAGs, task groups, or individual tasks.

Changelog

We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes.

Contributing Guide

All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.

A detailed overview an how to contribute can be found in the Contributing Guide.

As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct.

Goals for the project

• Goal 1

• Goal 2

• Goal 3

Limitations

• List any limitations

License

Apache License 2.0