Operators and Hooks for Airflow
Project description
airflow-plugin
Operators and Hooks for Airflow
Airflow at Earnest
We use heavily the KubernetesPodOperator
for the following reasons:
- Keep non-scheduling logic out of the scheduler.
- Avoid locking logic inside Airflow (containers can run through other schedulers).
- Avoid accumulating python dependencies in the Airflow image.
- Align with our engineering culture (functional programming and strongly typed languages lead to more maintainable code, Haskell preferred :) ).
As such, you will find most of the Operators we share inherit from the KubernetesPodOperator
.
Running the tests locally
-
Install dev requirements
pip install -r requirements_dev.txt
-
Run the unit tests
pytest -v -m "not kubernetes"
-
Start a local kubernetes cluster through docker-for-desktop (Mac/Windows) or minikube (Linux)
-
Bootstrap the cluster (it'll use the
airflow
namespace)tests/resources/kubernetes/bootstrap.sh
-
Run the kubernetes tests
AIRFLOW__KUBERNETES__IN_CLUSTER=False pytest -v -m "kubernetes"
Releasing
Currently, publishing a new version of the project is a manual process:
-
Install dev requirements
pip install -r requirements_dev.txt
-
Edit the
version
in setup.cfg -
Create a new git release:
# e.g. git tag -a v0.0.1
There's a simple template in RELEASE_NOTES_TEMPLATE.md for a release message.
-
Push the release:
git push origin v0.0.1
-
Create a source distribution:
python setup.py sdist bdist_wheel
-
Upload to PyPi
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for earnest-airflow-plugin-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c301f30c9065269b6b44db749cfdfce6b0195ee6d0037035f65c257ba665c05c |
|
MD5 | 2cd5dfac84939774a0c3fde7d91ffd8f |
|
BLAKE2b-256 | 262e800aa80b7933df7a9c675bf6beb87ae99aab21d1623de0a8099cc7c1a456 |
Hashes for earnest_airflow_plugin-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 46f0708c99a2447303f7b86ca664a3644c78cdfb861ff33dbe1b8601815d65a8 |
|
MD5 | d68212c9dd0f9b87fe4ebc158b8b48f4 |
|
BLAKE2b-256 | 72a4a3293485941b6329b4fcd44496d290484733638984c53807b69eed17860b |