Armada Airflow Operator
Project description
armada-airflow-operator
An Airflow operator for interfacing with the armada client
Background
Airflow is an open source project focused on orchestrating Direct Acylic Graphs (DAGs) across different compute platforms. To interface Airflow with Armada, you should use our armada operator.
Airflow
The airflow documentation was used for setting up a simple test server.
setup-local-airflow.sh demonstrates how to run airflow locally using Airflow's SequentialExecutor. This is only used for testing purposes.
Adding custom dags requires you to create a ~/airflow/dags folder and copying the dag files under examples in that location. This allows you to test the DAG in your airflow test server.
Examples
For documentation by example, see hello_armada.py or bad_armada.py.
Operator Documentation
Usage
The operator is available on PyPi
python3.8 -m venv armada38
source armada38/bin/activate
python3.8 -m pip install armada-airflow
Development
From the top level of the repo, you should run make airflow-operator
. This will generate proto/grpc files in the jobservice folder.
Airflow with the Armada operator can be run alongside the other Armada services via the localdev docker-compose environment. It is manually started in this way:
cd localdev
docker-compose up -d airflow
Airflow's web UI will then be accessible at http://localhost:8081 (login with admin/admin).
You can install the package via pip3 install third_party/airflow
.
You can use our tox file that streamlines development lifecycle. For development, you can install black, tox, mypy and flake8.
python3.8 -m tox -e py38
will run unit tests.
python3.8 -m tox -e format
will run a format check
python3.8 -m tox -e format-code
will run black on your code.
python3.8 -m tox -e docs
will generate a new sphinx doc.
Releasing the client
Armada-airflow releases are automated via Github Actions, for contributors with sufficient access to run them.
- Commit and merge a change to
third_party/airflow/pyproject.toml
raising the version number the appropriate amount. We are using semver for versioning. - Navigate to the airflow operator release workflow in Github workflows, click the "Run Workflow" button on the right side, and choose "master" as the branch to use the workflow from.
- Once the workflow has completed running, verify the new version of Armada client has been uploaded to PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for armada_airflow-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53e45783e6e7d3af426520d27f9f935e998c0a48a94c5d621c322a58e365eb6f |
|
MD5 | 76bcff3701d6468aaab1a90050f02918 |
|
BLAKE2b-256 | c6c1702bdb9cd275d35d4dc1a17e04f176bebf2132ab90868bd15ba9d03fbcb1 |