Marquez integration with Airflow
Project description
marquez-airflow
A library that integrates Airflow DAGs with Marquez for automatic metadata collection.
Status
This library is under active development at WeWork.
Requirements
Installation
$ pip install marquez-airflow
To install from source run:
$ python setup.py install
Usage
To use this library, the line from airflow import DAG
needs to be replaced by from marquez_airflow import DAG
, see example below:
from marquez_airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
DAG_NAME = 'my_DAG_name'
default_args = {
'marquez_location': 'github://data-dags/dag_location/',
'marquez_input_urns': ["s3://some_data", "s3://more_data"],
'marquez_output_urns': ["s3://output_data"],
'owner': ...,
'depends_on_past': False,
'start_date': ...,
}
dag = DAG(DAG_NAME, schedule_interval='*/10 * * * *',
default_args=default_args, description="yet another DAG")
run_this = DummyOperator(task_id='run_this', dag=dag)
run_this_too = DummyOperator(task_id='run_this_too', dag=dag)
run_this_too.set_upstream(run_this)
Contributing
See CONTRIBUTING.md for more details about how to contribute.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
marquez-airflow-0.2.0.tar.gz
(10.5 kB
view details)
File details
Details for the file marquez-airflow-0.2.0.tar.gz
.
File metadata
- Download URL: marquez-airflow-0.2.0.tar.gz
- Upload date:
- Size: 10.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.43.0 CPython/3.6.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a63908a2179e01e9185b8d9af4818a81ed5c77a3ba53bf829cabf2795aeeff1 |
|
MD5 | 2dc3a49dcd0b084be5b8595c1a7bb2f4 |
|
BLAKE2b-256 | 9c5064581d24958534e70ba7e79c5a227e0114873d4184c51e14d8f7ff41fda4 |