An Apache Airflow provider for Anomalo
Project description
Apache Airflow Provider for Anomalo
A set of native Airflow operators for Anomalo
Compatibility
These operators were created and tested with
- Python 3.8-3.10
- Airflow 2.3+
- Anomalo python client 0.0.7
Installation
pip install apache-airflow-providers-anomalo
You can validate that it is correctly installed by running airflow providers list
on the command line and seeing if apache-airflow-providers-anomalo
is a listed providers package.
Airflow Setup
From the airflow UI, go to Admin > Connections and hit the +
button at the top to add a new connection.
From the "Connection Type" drop down, select "Anomalo".
Then fill in the fields for "Connection Id" (anomalo-default
is the default connection id), "Host", and "API Secret Token".
Usage
-
Obtain Anomalo table name from GUI. For example would be
public-bq.covid19_nyt.us_counties
-
This package includes 3 different operators. You can find documentation for them on the operator code itself.
- Run checks Operator:
airflow.providers.anomalo.operators.anomalo.AnomaloRunCheckOperator
- Job Sensor
airflow.providers.anomalo.sensors.anomalo.AnomaloJobCompleteSensor
- Validate table checks:
airflow.providers.anomalo.operators.anomalo.AnomaloPassFailOperator
- Run checks Operator:
-
See
anomalo_dag_example.py
for usage example
Releasing to PyPi
To release a new version to PyPi, you will need to
- Configure your pypi token by running:
poetry config pypi-token.pypi [token here]
The token can be found in the AWS secrets manager - Bump the version number in pyproject.toml. Make sure that this change is committed.
- run
poetry publish --build
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apache_airflow_providers_anomalo-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d835a6e748dc44de1a81366d1fc9953c93060ea461afe0666798cc2cbf748748 |
|
MD5 | 97e34f95a6226e9bda3916d8248996d3 |
|
BLAKE2b-256 | 0d0436478b7b2edaf88a53757a8cd74cc4d2d1f295e43641d10e2eae3c8984d2 |
Hashes for apache_airflow_providers_anomalo-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f31a46ea88bb9e5add43f7ace80020f0a0e6eac478ab63e0384a677fc2241f4 |
|
MD5 | 8119927cca7499bb088220ddb91e2e7f |
|
BLAKE2b-256 | 96ff33f12f4c63030715023e5c0e0bbe6d3fcdd7f7ed8df2bac2f43ef2ef93b7 |