Marquez integration with Airflow
Project description
marquez-airflow
Is a library that integrates Airflow DAGs with Marquez for automatic metadata collection.
Requirements
- Python 3.5+
- apache-airflow 1.10.0+
- marquez-client
Installation
pip install marquez-airflow
Usage
Once the library is installed in your system, your current DAGs need to be modified slightly by changing the DAG to a MarquezDAG, see example below:
from marquez.airflow import MarquezDag as DAG
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime
DAG_NAME = 'my_DAG_name'
default_args = {
'mqz_namespace': 'namespace_1',
'mqz_location': 'github://data-dags/dag_location/',
'mqz_input_datasets': ["s3://some_data", "s3://more_data"],
'mqz_output_datasets': ["s3://output_data"],
'owner': ...,
'depends_on_past': False,
'start_date': ...,
}
dag = DAG(DAG_NAME, schedule_interval='*/10 * * * *',
default_args=default_args, description="yet another DAG")
run_this = DummyOperator(task_id='run_this', dag=dag)
run_this_too = DummyOperator(task_id='run_this_too', dag=dag)
run_this_too.set_upstream(run_this)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
marquez-airflow-0.0.2.tar.gz
(2.6 kB
view hashes)
Built Distribution
Close
Hashes for marquez_airflow-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6dc4baf9d18c2e64968d30268573b8217bfeef479f26f2079e4b4cff0b35e1f4 |
|
MD5 | c8087b0c4d8fde19388cd7ff5b7f71b8 |
|
BLAKE2b-256 | 19636226a3c6ea025de5abe417ce6c0fd775e8e394848dcd2e03016fe117fea3 |