otb-airflow-triggers

Introduction

This repository contains custom openthebox Airflow triggers for the openthebox Airflow repository.

The triggers must be available to the Airflow triggerer service as well as the worker pods. They therefore need a separate package that each service can install independently. Code available only in worker pods cannot run in the triggerer service.

Dependencies

The dependencies must target the same versions as those in the airflow-dag repository. Keep the versions in pyproject.toml aligned with that repository when dependencies change. This includes Airflow and the Amazon provider. The Amazon provider includes its aiobotocore extra for asynchronous AWS calls.

Local development

Use Python 3.12 or later. Run these commands from the project root to create and activate a virtual environment. Install the package with its development dependencies.

python3.12 -m venv venv
source venv/bin/activate
python -m pip install -e '.[dev]'

Run these checks from the project root.

venv/bin/ruff check .
venv/bin/ruff format --check .
venv/bin/mypy src/ tests/
venv/bin/bandit -r src/
venv/bin/coverage run -m pytest
venv/bin/coverage report

Build

The package version comes from Git tags, such as v0.1.0. Commits after a release tag produce a development version. Git hash suffixes are omitted so that public package indexes accept development versions. The generated version file is included in the source archive.

Build the wheel and source archive.

venv/bin/python -m build
venv/bin/python -m twine check --strict dist/*

The build writes the wheel and source archive to dist/. Install the package in each Airflow environment that imports its triggers. This includes the worker pods and the triggerer service.

GitHub workflows

Pull requests into main run the quality checks in ci.yml. The checks cover tests, coverage, lint rules, formatting, and types. They also check dependency compatibility, security issues with Bandit, and package metadata. The checks run with Python 3.12.

Publishing a GitHub release starts release.yml. It checks the release tag, builds the package, and publishes it to PyPI. It then attaches the wheel and source archive to the existing GitHub release. The package version must match the tag and the GitHub pre-release setting. For example, tag v0.1.0rc1 must have the pre-release setting enabled.

The Publish release candidate workflow supports manual publishing from a branch or tag. It runs the same quality checks and builds the exact commit that passed them. It publishes a release candidate to PyPI or TestPyPI. It does not create a GitHub release or attach files to a release. Workflow artifacts transfer distributions between jobs and expire after one day.

Publishing setup

Create the pypi and testpypi environments in the GitHub repository settings. Configure these Trusted Publishers for the otb-airflow-triggers project. Use owner openthebox and repository airflow-triggers for each publisher.

Trusted Publishing uses GitHub's identity token and does not need stored PyPI API tokens.

Manual publishing

The workflow must exist on the default branch before GitHub makes manual runs available.

1. Open the repository's Actions page.
2. Select Publish release candidate.
3. Select Run workflow.
4. Select the branch or tag under Use workflow from.
5. Select pypi or testpypi in target.
6. Start the workflow.

The version comes from hatch-vcs and the selected commit's Git history. For example, commits after v0.1.0 produce a version such as 0.1.1.dev7. A tag such as v0.1.1rc1 produces 0.1.1rc1. Manual publishing requires a pre-release version and rejects final-release tags. Each package index accepts a version only once. Select a later commit or a new pre-release tag for another upload.