Operators and Hooks for Airflow
Project description
airflow-plugin
Operators and Hooks for Airflow
Airflow at Earnest
We use heavily the KubernetesPodOperator
for the following reasons:
- Keep non-scheduling logic out of the scheduler.
- Avoid locking logic inside Airflow (containers can run through other schedulers).
- Avoid accumulating python dependencies in the Airflow image.
- Align with our engineering culture (functional programming and strongly typed languages lead to more maintainable code, Haskell preferred :) ).
As such, you will find most of the Operators we share inherit from the KubernetesPodOperator
.
Running the tests locally
-
Install dev requirements
pip install -r requirements_dev.txt
-
Run the unit tests
pytest -v -m "not kubernetes"
-
Start a local kubernetes cluster through docker-for-desktop (Mac/Windows) or minikube (Linux)
-
Bootstrap the cluster (it'll use the
airflow
namespace)tests/resources/kubernetes/bootstrap.sh
-
Run the kubernetes tests
AIRFLOW__KUBERNETES__IN_CLUSTER=False pytest -v -m "kubernetes"
Releasing
Currently, publishing a new version of the project is a manual process:
-
Install dev requirements
pip install -r requirements_dev.txt
-
Edit the
version
in setup.cfg -
Create a new git release:
# e.g. git tag -a v0.0.1
There's a simple template in RELEASE_NOTES_TEMPLATE.md for a release message.
-
Push the release:
git push origin v0.0.1
-
Create a source distribution:
python setup.py sdist bdist_wheel
-
Upload to PyPi
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for earnest-airflow-plugin-0.0.2b3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a858511b6cc9b6d44e59c53b81dd8c034eac89d50eb0462beceed924c6ead0ac |
|
MD5 | 797ef16b89c3a3584e477a4e1a8a93db |
|
BLAKE2b-256 | 5fbfdf747241be74d2e6151b537bc7cba6b7cc995dad7fae9449455dab035915 |
Hashes for earnest_airflow_plugin-0.0.2b3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc54f91c8e7bf4bd6f3b38ace94b3812a8fee85a6a3a246ec012b0471c8ba804 |
|
MD5 | 0d1be87293893bfbefee6a6825714f21 |
|
BLAKE2b-256 | c7950ed3c4930aedc5c6d1a923f8a869316cf5a8b09d22614d51659a5377cc4a |