otb-airflow-triggers

Introduction

This repository contains custom openthebox Airflow triggers for the openthebox Airflow repository.

The triggers must be available to the Airflow triggerer service as well as the worker pods. They therefore need a separate package that each service can install independently. Code available only in worker pods cannot run in the triggerer service.

Dependencies

The dependencies must target the same versions as those in the airflow-dag repository. Keep the versions in pyproject.toml aligned with that repository when dependencies change. This includes Airflow and the Amazon provider. The Amazon provider includes its aiobotocore extra for asynchronous AWS calls.

Triggers

Local development

Use Python 3.12 or later. Run these commands from the project root to create and activate a virtual environment. Install the package with its development dependencies.

# Create a virtual environment.
python3.12 -m venv venv
# Activate the virtual environment.
source venv/bin/activate
# Install the package and its development dependencies.
python -m pip install -e '.[dev]'

Run these checks from the project root.

# Check the lint rules.
ruff check .
# Check the code format.
ruff format --check .
# Check the types.
mypy src/ tests/
# Check the source code for security issues.
bandit -r src/
# Run the tests and collect coverage data.
coverage run -m pytest
# Show the coverage report.
coverage report

GitHub workflows

Pull requests into main run the quality checks in ci.yml. The checks cover tests, coverage, lint rules, formatting, and types. They also check dependency compatibility, security issues with Bandit, and package metadata. The checks run with Python 3.12.

Deployment

Production release

Create and publish a GitHub release to start the full production release workflow, release.yml. It runs the quality checks, checks the release tag, builds the package, and publishes it to PyPI. It then attaches the wheel and source archive to the existing GitHub release. The package version must match the tag and the GitHub pre-release setting. For example, tag v0.1.0rc1 must have the pre-release setting enabled.

Release candidate

The Publish release candidate workflow supports manual publishing from a branch or tag. It runs the same quality checks and builds the exact commit that passed them. It publishes a release candidate to PyPI or TestPyPI. It does not create a GitHub release or attach files to a release. Workflow artifacts transfer distributions between jobs and expire after one day.

Start this workflow manually and select the package index in the target input. The workflow must exist on the default branch before GitHub makes manual runs available.

1. Open the repository's Actions page.
2. Select Publish release candidate.
3. Select Run workflow.
4. Select the branch or tag under Use workflow from.
5. Select pypi or testpypi in target.
6. Start the workflow.

The version comes from hatch-vcs and the selected commit's Git history. For example, commits after v0.1.0 produce a version such as 0.1.1.dev7. A tag such as v0.1.1rc1 produces 0.1.1rc1. Manual publishing requires a pre-release version and rejects final-release tags. Each package index accepts a version only once. Select a later commit or a new pre-release tag for another upload.