Marquez integration with Airflow
Project description
marquez-airflow
A library that integrates Airflow DAGs
with Marquez for automatic metadata collection.
Features
Metadata
- Task lifecycle
- Task parameters
- Task runs linked to versioned code
- Task inputs / outputs
Lineage
- Track inter-DAG dependencies
Built-in
- SQL parser
- Link to code builder (ex: GitHub)
- Metadata extractors
Requirements
Installation
$ pip3 install marquez-airflow
Note: You can also add
marquez-airflow
to yourrequirements.txt
for Airflow.
To install from source, run:
$ python3 setup.py install
Configuration
The library depends on a backend. A Backend
is configurable and lets the library know where to write dataset, job, and run metadata.
Backends
HTTP
: Write metadata to MarquezFILE
: Write metadata to a file (asjson
) under/tmp/marquez
LOG
: Simply just logs the metadata to the console
By default, the HTTP
backend will be used (see next sections on configuration). To override the default backend and write metadata to a file, use MARQUEZ_BACKEND
:
MARQUEZ_BACKEND=FILE
Note: Metadata will be written to
/tmp/marquez/client.requests.log
, but the location can be overridden withMARQUEZ_FILE
.
HTTP
Backend Authentication
The HTTP
backend supports using API keys to authenticate requests via Bearer
auth. To include a key when making an API request, use MARQUEZ_API_KEY
:
MARQUEZ_BACKEND=HTTP
MARQUEZ_API_KEY=[YOUR_API_KEY]
HTTP
Backend Environment Variables
marquez-airflow
needs to know where to talk to the Marquez server API. You can set these using environment variables to be read by your Airflow service.
You will also need to set the namespace if you are using something other than the default
namespace.
MARQUEZ_BACKEND=HTTP
MARQUEZ_URL=http://my_hosted_marquez.example.com:5000
MARQUEZ_NAMESPACE=my_special_ns
Extractors : Sending the correct data from your DAGs
If you do nothing, Marquez will receive the Job
and the Run
from your DAGs, but sources and datasets will not be sent.
marquez-airflow
allows you to do more than that by building "Extractors". Extractors are in the process of changing right now, but they basically take a task and extract:
- Name : The name of the task
- Location : Location of the code for the task
- Inputs : List of input datasets
- Outputs : List of output datasets
- Context : The Airflow context for the task
It's important to understand the inputs and outputs are lists and relate directly to the Dataset
object in Marquez. Datasets also include a source which relates directly to the Source
object in Marquez.
Usage
To begin collecting Airflow DAG metadata with Marquez, use:
- from airflow import DAG
+ from marquez_airflow import DAG
When enabled, the library will:
- On DAG start, collect metadata for each task using an
Extractor
(the library defines a default extractor to use otherwise) - Collect task input / output metadata (
source
,schema
, etc) - Collect task run-level metadata (execution time, state, parameters, etc)
- On DAG complete, also mark the task as complete in Marquez
To enable logging, set the environment variable MARQUEZ_LOG_LEVEL
to DEBUG
, INFO
, or ERROR
:
$ export MARQUEZ_LOG_LEVEL=INFO
Development
To install all dependencies for local development:
$ pip3 install -e .[dev]
To run the entire test suite, you'll first want to initialize the Airflow database:
$ airflow initdb
Then, run the test suite with:
$ pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file marquez-airflow-0.14.0.tar.gz
.
File metadata
- Download URL: marquez-airflow-0.14.0.tar.gz
- Upload date:
- Size: 26.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bdbec23da81c340765464231fe2ca4ce3f226138bae28ca80ef0182617ad6e9 |
|
MD5 | cdc816e4c439f7cab61db0d84cf815b8 |
|
BLAKE2b-256 | 24bb521d0902b3ad123f5ac4ecf33e97f20d2511294b8098e36409c4da698d74 |