Skip to main content

Marquez integration with Airflow

Project description

marquez-airflow

CircleCI codecov status Gitter version license

A library that integrates Airflow DAGs with Marquez for automatic metadata collection.

Status

This library is under active development at WeWork.

Requirements

Installation

$ pip install marquez-airflow

To install from source run:

$ python setup.py install

Usage

To use this library, the line from airflow import DAG needs to be replaced by from marquez_airflow import DAG, see example below:

from marquez_airflow import DAG
from airflow.operators.dummy_operator import DummyOperator


DAG_NAME = 'my_DAG_name'

default_args = {
    'marquez_location': 'github://data-dags/dag_location/',
    'marquez_input_urns': ["s3://some_data", "s3://more_data"],
    'marquez_output_urns': ["s3://output_data"],
    
    'owner': ...,
    'depends_on_past': False,
    'start_date': ...,
}

dag = DAG(DAG_NAME, schedule_interval='*/10 * * * *',
          default_args=default_args, description="yet another DAG")

run_this = DummyOperator(task_id='run_this', dag=dag)
run_this_too = DummyOperator(task_id='run_this_too', dag=dag)
run_this_too.set_upstream(run_this)

Contributing

See CONTRIBUTING.md for more details about how to contribute.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

marquez-airflow-0.2.0.tar.gz (10.5 kB view details)

Uploaded Source

File details

Details for the file marquez-airflow-0.2.0.tar.gz.

File metadata

  • Download URL: marquez-airflow-0.2.0.tar.gz
  • Upload date:
  • Size: 10.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.43.0 CPython/3.6.10

File hashes

Hashes for marquez-airflow-0.2.0.tar.gz
Algorithm Hash digest
SHA256 4a63908a2179e01e9185b8d9af4818a81ed5c77a3ba53bf829cabf2795aeeff1
MD5 2dc3a49dcd0b084be5b8595c1a7bb2f4
BLAKE2b-256 9c5064581d24958534e70ba7e79c5a227e0114873d4184c51e14d8f7ff41fda4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page