An Apache Airflow provider for Anomalo
Project description
Apache Airflow Provider for Anomalo
A set of native Airflow operators for Anomalo
Compatibility
These operators were created and tested with
- Python 3.8-3.10
- Airflow 2.3+
- Anomalo python client 0.0.7
Installation
pip install apache-airflow-providers-anomalo
You can validate that it is correctly installed by running airflow providers list
on the command line and seeing if apache-airflow-providers-anomalo
is a listed providers package.
Airflow Setup
From the airflow UI, go to Admin > Connections and hit the +
button at the top to add a new connection.
From the "Connection Type" drop down, select "Anomalo".
Then fill in the fields for "Connection Id" (anomalo-default
is the default connection id), "Host", and "API Secret Token".
Usage
-
Obtain Anomalo table name from GUI. For example would be
public-bq.covid19_nyt.us_counties
-
This package includes 3 different operators. You can find documentation for them on the operator code itself.
- Run checks Operator:
airflow.providers.anomalo.operators.anomalo.AnomaloRunCheckOperator
- Job Sensor
airflow.providers.anomalo.sensors.anomalo.AnomaloJobCompleteSensor
- Validate table checks:
airflow.providers.anomalo.operators.anomalo.AnomaloPassFailOperator
- Run checks Operator:
-
See
anomalo_dag_example.py
for usage example
Releasing to PyPi
To release a new version to PyPi, you will need to
- Configure your pypi token by running:
poetry config pypi-token.pypi [token here]
The token can be found in the AWS secrets manager - Bump the version number in pyproject.toml. Make sure that this change is committed.
- run
poetry publish --build
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache_airflow_providers_anomalo-0.2.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d744e8157ffc887a74b9487424fa26e5c39eb0fc94b79a2075f0a68962020a1 |
|
MD5 | 4babe5ce68f4c24d2bfff613c8fdc445 |
|
BLAKE2b-256 | 63e026c0af3227f631d508fdf448ff212406802e4bf9504de2408aed1946cca0 |
Hashes for apache_airflow_providers_anomalo-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 445bf1817a461cbfb2028023eec3c2fdc18963ca764364f3d264c1b0f3ba357e |
|
MD5 | 8c28005629a0deb709ebf3b480187842 |
|
BLAKE2b-256 | 9b5faba1ed2a81087803b8af2e997d25d6a10b2de644bb11343cf63da8d20f3a |