Skip to main content

Execute pipeline stages and steps

Project description

Staging

Build PyPI Code style: black pylint NoPrint

About

Automate your commands - be it for environment setup or in CI/CD pipeline. Parallelize execution, and make it easier to handle dependent commands.

Requirements

This package requires tomlkit and dataclass-wizard packages.

Usage

Full description

Staging is configurable in pyproject.toml file and is accessible through CLI interface.

In order to use staging, you have to define steps and then stages. Stages are built from steps.

Let's see example below (which uses all available options):

[tool.staging.steps]
# Test
get_test_packages = {execute = "command to get test packages", output="test_packages"}
coverage = { prepare = "coverage run -m pytest -xv {packages}", execute = "coverage report {flags}", cleanup = "coverage erase", output = "coverage_result", format = {flags = "flags", packages= "test_packages"} }
# Lint
clean = {execute = "command to execute", success_codes=[0,1,2,3,4,5]}
isort = { execute = "isort {flags} . tests", format = {flags = "flags"}}
black = { execute = "black {flags} . tests", format = {flags = "flags"}}
pylint = { execute = "pylint {package} tests", format = {package = "package"}}
noprint = { execute = "noprint -ve {package} tests", format = {package = "package"}, error_codes=[1,2]}
finish = { execute = "finishing command" }

[tool.staging.stages.format]
description = "Format code"
format = {package="staging"}
steps = [
    {step = "isort"},
    {step = "black"},
]

[tool.staging.stages.lint]
description = "Check linting"
format = {flags="--check", package="staging"}
steps = [
    {step = "clean", continue_on_failure=true},
    {parallel = {steps = ["isort", "black", "pylint", "noprint"]}, continue_on_failure=true},
    {step = "finish"},
]

[tool.staging.stages.test]
description = "Test the package"
format = {flags="-m --fail-under=30"}
steps = [
    {step = "get_test_packages"},
    {step = "coverage"},
]

Here we have defined 7 different steps: get_test_packages, coverage, clean, and so on. These 7 steps are later used in 3 stages: test, lint, and format.

Let's say we execute staging format. What happens?

  1. Stage context is updated with formatter {key=package, value=staging}.
  2. Step isort is executed. Command isort {flags} . tests is formatted using stage formatter. Since flags is empty, the final command is simply isort . tests.
  3. Step black is executed. This works the same as isort above.

Now let's say we execute staging lint.

  1. Context has formatter for value package, but also for flags.
  2. Step clean is executed. It might fail, but we don't care and instead the process is continued.
  3. Steps isort, black, pylint and noprint are all executed in a thread pool due to being specified in a parallel block. This time however, even though some of the steps are the same as in the staging format, we now have a flags formatter defined. Thanks to this, we now have executed isort --check . tests command that would verify if the formatting was properly applied. Due to how steps are configured - specified custom success and error codes - the parallel block would fail (one command will crash others). However, the parallel block allows to specify continue_on_error as well, just like step block.
  4. As such, finish block is also executed.

Last but not least, staging test.

  1. Similarly to previous stages, we set up the stage formatter context.
  2. Step get_test_packages outputs a package name to stdout. This value will be saved under test_packages key in the staging context.
  3. Step coverage has a bit of a different structure, it has prepare and cleanup blocks on top of typical execute. Prepare is executed before execute. However, it's not taken into account when writing to output. Moreover, it will always fail when status code is different to 0. Block called cleanup is just like what it says it is. It's a cleanup process executed after execute. It will be executed no matter what is the final status of the execute command.

You can also chain your stages in one command. Keep in mind however that formatting contexts are defined on per stage basis. So if you execute staging format lint - formatting context from stage format won't be propagated to stage lint. Running stages in one execution is also performed in serial manner - no parallelism.

Example

For real-life examples, see pyproject.toml file for this package.

Development

Installation

Install virtual environment and check_bump package in editable mode with dev dependencies.

python -m venv venv
source venv/bin/activate
pip install -e .[dev]

How to?

Automate as much as we can, see configuration in pyproject.toml file to see what are the flags used.

staging format  # Reformat the code
staging lint    # Check for linting issues
staging test    # Run unit tests and coverage report

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

staging-0.0.2.tar.gz (10.8 kB view details)

Uploaded Source

Built Distribution

staging-0.0.2-py3-none-any.whl (9.3 kB view details)

Uploaded Python 3

File details

Details for the file staging-0.0.2.tar.gz.

File metadata

  • Download URL: staging-0.0.2.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for staging-0.0.2.tar.gz
Algorithm Hash digest
SHA256 f5e0c75c870a659e32d9d4fede3c6a23b4d547f82f30b2910be41bd902471c17
MD5 1f96fa5bf106c1f4b3e8cf75d75564db
BLAKE2b-256 22fb1c4bd619dde709edd1647658d47dc159dc889287f2cc21b476cd3eda7eef

See more details on using hashes here.

File details

Details for the file staging-0.0.2-py3-none-any.whl.

File metadata

  • Download URL: staging-0.0.2-py3-none-any.whl
  • Upload date:
  • Size: 9.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for staging-0.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 7ed1ac96e566d2aeb6f336c9466d129a3b2f24a238c0fbee0ebacda96067c26c
MD5 b25639d3cc4c23dee07e196ad365abd9
BLAKE2b-256 c5dbca8e01894f3b3bfa6d267d329324a1fa41137aede94e701b63708643c5a1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page