Operators and Hooks for Airflow
Project description
airflow-plugin
Operators and Hooks for Airflow
Airflow at Earnest
We use heavily the KubernetesPodOperator
for the following reasons:
- Keep non-scheduling logic out of the scheduler.
- Avoid locking logic inside Airflow (containers can run through other schedulers).
- Avoid accumulating python dependencies in the Airflow image.
- Align with our engineering culture (functional programming and strongly typed languages lead to more maintainable code, Haskell preferred :) ).
As such, you will find most of the Operators we share inherit from the KubernetesPodOperator
.
Running the tests locally
-
Install dev requirements
pip install -r requirements_dev.txt
-
Run the unit tests
pytest -v -m "not kubernetes"
-
Start a local kubernetes cluster through docker-for-desktop (Mac/Windows) or minikube (Linux)
-
Bootstrap the cluster (it'll use the
airflow
namespace)tests/resources/kubernetes/bootstrap.sh
-
Run the kubernetes tests
AIRFLOW__KUBERNETES__IN_CLUSTER=False pytest -v -m "kubernetes"
Releasing
Currently, publishing a new version of the project is a manual process:
-
Install dev requirements
pip install -r requirements_dev.txt
-
Edit the
version
in setup.cfg -
Create a new git release:
# e.g. git tag -a v0.0.1
There's a simple template in RELEASE_NOTES_TEMPLATE.md for a release message.
-
Push the release:
git push origin v0.0.1
-
Create a source distribution:
python setup.py sdist bdist_wheel
-
Upload to PyPi
twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for earnest-airflow-plugin-0.0.2b4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d839cf10c9681589a6f720183154c6acad01dc7dd91f34e6d79ec25e56a501b9 |
|
MD5 | 2228c6f4ce6afe30f7b06fe3dd1a9c59 |
|
BLAKE2b-256 | 61089fa3b2a9ca873eaaf50930dd65b500fe31cb4d08af94c20a035f968639c4 |
Hashes for earnest_airflow_plugin-0.0.2b4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3825e8d4bda2388c6705fe54042e3479a751a8ee159703fea9f0bea55137281a |
|
MD5 | fbfe1ce560fe272004111cc5ad81027a |
|
BLAKE2b-256 | ace23dcfb9c040bbf8ceecff9802af297e3184157800240fb20ce5f50786b6b7 |