Operators and Hooks for Airflow
Project description
airflow-plugin
Operators and Hooks for Airflow
Airflow at Earnest
We use heavily the KubernetesPodOperator
for the following reasons:
- Keep non-scheduling logic out of the scheduler.
- Avoid locking logic inside Airflow (containers can run through other schedulers).
- Avoid accumulating python dependencies in the Airflow image.
- Align with our engineering culture (functional programming and strongly typed languages lead to more maintainable code, Haskell preferred :) ).
As such, you will find most of the Operators we share inherit from the KubernetesPodOperator
.
Running the tests locally
-
Install dev requirements
pip install -r requirements_dev.txt
-
Run the unit tests
pytest -v -m "not kubernetes"
-
Start a local kubernetes cluster through docker-for-desktop (Mac/Windows) or minikube (Linux)
-
Bootstrap the cluster (it'll use the
airflow
namespace)tests/resources/kubernetes/bootstrap.sh
-
Run the kubernetes tests
AIRFLOW__KUBERNETES__IN_CLUSTER=False pytest -v -m "kubernetes"
Releasing
Currently, publishing a new version of the project is a manual process:
-
Install dev requirements
pip install -r requirements_dev.txt
-
Edit the
version
in setup.cfg -
Create a new git release:
# e.g. git tag -a v0.0.1
There's a simple template in RELEASE_NOTES_TEMPLATE.md for a release message.
-
Push the release:
git push origin v0.0.1
-
Create a source distribution:
python setup.py sdist bdist_wheel
-
Upload to PyPi
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for earnest-airflow-plugin-0.0.2b1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b92ee791abde86c0252581e82a9b1727dcd6f008f6e3565e0bd9869fd4e43c3e |
|
MD5 | 6d2d8702724729767f43cb0afa35fd23 |
|
BLAKE2b-256 | 31a5ec891bf0e6bdc464df5f3d8904020e8377063a631840ab66e425296a7dd0 |
Hashes for earnest_airflow_plugin-0.0.2b1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e816e6f26befa45ef60ee3ca877e7c3c232fa66c386a1a0d9be75fa8571fe07d |
|
MD5 | 246fc7ed87d4b30d7777028cda3778f3 |
|
BLAKE2b-256 | 3df36f0aed001981602ca53c310277d335c07b7918fd78838162ca817641de1d |