Skip to main content

Airflow plugins for Rekcurd

Project description

# Rekcurd-airflow
[![Build Status](https://travis-ci.com/rekcurd/airflow-plugin.svg?branch=master)](https://travis-ci.com/rekcurd/airflow-plugin)
[![PyPI version](https://badge.fury.io/py/rekcurd-airflow.svg)](https://badge.fury.io/py/rekcurd-airflow)
[![codecov](https://codecov.io/gh/rekcurd/airflow-plugin/branch/master/graph/badge.svg)](https://codecov.io/gh/rekcurd/airflow-plugin "Non-generated packages only")
[![pypi supported versions](https://img.shields.io/pypi/pyversions/rekcurd-airflow.svg)](https://pypi.python.org/pypi/rekcurd-airflow)


Airflow Plugins for [Rekcurd Dashboard](https://github.com/rekcurd/dashboard)

## Environment
- Python 3.6
- apache-airflow >= 1.10.x
- Rekcurd Dashboard >= 0.4.x

## Setup
- Setup Airflow
- Setup Rekcurd Dashboard and get your access token
- Run the following commands

**Use JWT token published by Rekcurd Dashboard as REKCURD_ACCESS_TOKEN for now**

```
# Set the access token and airflow-rekcurd connection to airflow
# Replace the environment variables with your own values.
$ cd $AIRFLOW_HOME
$ airflow variables -s rekcurd_access_token $REKCURD_ACCESS_TOKEN
$ airflow connections -a --conn_id rekcurd_dashboard \
--conn_uri http://$REKCURD_DASHBOARD_HOST:$REKCURD_DASHBOARD_PORT/

$ pip install rekcurd-airflow
```

## Components
You can see example DAGs for the plugins in [here](./dags)

### Operators
- EvaluationUpdateOperator. ([Example DAG](./dags/example_evaluation_upload.py))
- ModelDeleteOperator. ([Example DAG](./dags/example_model_delete.py))
- ModelEvaluateOperator. ([Example DAG](./dags/example_model_evaluate.py))
- ModelSwitchOperator. ([Example DAG](./dags/example_model_switch.py))
- ModelUpdateOperator. ([Example DAG](./dags/example_model_upload.py))

[Overall Example DAG](./dags/example_all.py)

## Test

```
$ cd airflow-plugin # move to the root directory of this repo

# not necessary if you already setup airflow for this repository
$ export AIRFLOW_HOME=`pwd`
$ pip install -r test-requirements.txt
$ airflow initdb
$ fernet_key=`python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`
$ sed -i -e "s/\(fernet_key =\).*/\1 $fernet_key/" airflow.cfg

$ python -m unittest
```


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rekcurd-airflow, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size rekcurd_airflow-0.1.0-py3-none-any.whl (13.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size rekcurd-airflow-0.1.0.tar.gz (5.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page