CI/Python

This project contains GitLab CI templates and components that we can use in GitLab CI pipelines of Python projects.

[TOC]

Components

build-wheel

Component that builds a wheel and tarball for the given project (it runs python3 -m build) using the provided python-version. In addition, the component checks that the wheel does not contain the tests/ directory, and that the files with the files-not-in-dist regex pattern are not present either in the produced wheel or tarball.

Example:

include:
  - component: "gitlab.desy.de/fs-sc/ci/python/build-wheel@main"
    inputs:
      python-version: "3.10"
      files-not-in-dist: "(cspell|etc/|templates/)"

pre-commit

Component that runs pre-commit using the provided python-versions. You can install packages before pre-commit runs by providing install-commands a list of commands to execute. Typical install commands are python3 -m pip install <package> and apt-get install <package>.

Example:

include:
  - component: "gitlab.desy.de/fs-sc/ci/python/pre-commit@main"
    inputs:
      python-versions: ["3.10"]
      install-commands:
        - apt-get install -y --no-install-recommends libatomic1
        - python3 -m pip install mypy

publish-wheel

Component that publishes the wheel and tarball produced by the build-wheel component to PyPI by running the twine upload command with the provided python-version and twine-options, the default provided twine-options being --password "$PYPI_API_TOKEN" --skip-existing. The component only runs for Git tags.

Example:

include:
  - component: "gitlab.desy.de/fs-sc/ci/python/publish-wheel@main"
    inputs:
      python-version: "3.10"
      # Assuming that you already defined PYPI_API_TOKEN:
      twine-options: --password "$PYPI_API_TOKEN" --skip-existing

pytest, pytest-tarball, pytest-wheel, pytest-with-coverage

The pytest component runs pytest using the provided python-versions and pytest-options. You can install packages before pytest runs by providing install-commands a list of commands to execute. Typical install commands are python3 -m pip install <package> and apt-get install <package>. You can also provide the CI job with services that will run Docker containers accessible from the tests.

Example:

include:
  - component: "gitlab.desy.de/fs-sc/ci/python/pytest@main"
    inputs:
      python-versions: ["3.10", "3.11", "3.12", "3.13", "3.14"]
      install-commands:
        - python3 -m pip install "${CI_PROJECT_DIR}[asapo,tests,zmq]"
      pytest-options: "-vv --log-cli-level=DEBUG"
      services:
        - name: gitlab.desy.de:5555/asapo/asapo/asapo-standalone-dev:latest

Similarly, the components pytest-tarball and pytest-wheel runs pytest respectively on the tarball and wheel produced by the build-wheel component.

The component pytest-with-coverage runs pytest with coverage and produces a JUnit XML test report. This component accepts a single python-version instead of several python-versions for the other components.

To avoid repeating the code between the includes of these components, we can extract a YAML anchor. For example:

.common_test_inputs: &common_test_inputs
  install-commands:
    - python3 -m pip install "${PACKAGE}[asapo,tests,zmq]"
  pytest-options: "-vv --log-cli-level=DEBUG"
  services:
    - name: gitlab.desy.de:5555/asapo/asapo/asapo-standalone-dev:latest

include:
  - component: "gitlab.desy.de/fs-sc/ci/python/build-wheel@main"
  - component: "gitlab.desy.de/fs-sc/ci/python/pytest@main"
    inputs:
      <<: *common_test_inputs
      python-versions: ["3.9"]
  - component: "gitlab.desy.de/fs-sc/ci/python/pytest-with-coverage@main"
    inputs:
      <<: *common_test_inputs
      python-version: "3.9"
  - component: "gitlab.desy.de/fs-sc/ci/python/pytest-wheel@main"
    inputs:
      <<: *common_test_inputs
      python-versions: ["3.9", "3.10", "3.11", "3.12"]
  - component: "gitlab.desy.de/fs-sc/ci/python/pytest-tarball@main"
    inputs:
      <<: *common_test_inputs
      python-versions: ["3.9"]

Templates

templates/workflows/branch-pipelines.yml

You can use the branch-pipelines workflow template to allow CI pipelines to run only on branches and tags with the format v<PEP 440 version>, see PEP_440_VERSION_REGEX.

Usage example:

include:
  - project: "fs-sc/ci/python"
    ref: "main"
    file:
      - "templates/workflows/branch-pipelines.yml"